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Abstract 


r  provides  guidance  for  use 
porting  organizations  for  fulfilling  the  objectives  of 


by  Government  personnel  and  their  sup- 
the  SCAMPI  method  in  their  acquisition 


The  Standard  CMMI®  Appraisal  Method  for  Process  Improvement  (SCAMPIsm)  is  designed 
to  provide  benchmark  quality  ratings  relative  to  Capability  Maturity  Model®  Integration 
(CMMI)  models.  It  is  applicable  to  a  wide  range  of  appraisal  usage  modes,  including  both 
internal  process  improvement  and  external  capability  determinations.  SCAMPI  satisfies  all  of 
the  Appraisal  Requirements  for  CMMI  (ARC)  requirements  for  a  Class  A  appraisal  method 
and  can  support  the  conduct  of  ISO/IEC  15504  assessments.  The  SCAMPI  Method  Defini¬ 
tion  Document  describes  the  requirements,  activities,  and  practices  associated  with  each  of 
the  processes  that  compose  the  SCAMPI  method.  It  is  intended  to  be  one  of  the  elements  of 
the  infrastructure  within  which  SCAMPI  Lead  Appraisers  conduct  a  SCAMPI  appraisal.  Pre¬ 
cise  listings  of  required  practices,  parameters,  and  variation  limits,  as  well  as  optional  prac¬ 
tices  and  guidance  for  enacting  the  method,  are  covered.  An  overview  of  the  method’s  con¬ 
text,  concepts,  and  architecture  is  also  provided. 


®  CMMI  and  Capability  Maturity  Model  are  registered  in  the  U.S.  Patent  and  Trademark  Office  by 
Carnegie  Mellon  University. 

SM  SCAMPI  and  CMM  Integration  are  service  marks  of  Carnegie  Mellon  University. 
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About  This  Document 


The  SCAMPI  VI.  1  Method  Implementation  Guidance  for  Government  Source  Selection  and 
Contract  Process  Monitoring  (MIG-GSS/CPM)  document  provides  guidance  that  supple¬ 
ments  the  usage  of  the  SCAMPI  method  in  Government  acquisition  environments,  The 
document  wholly  incorporates  the  SCAMPI  VI.  1  Method  Definition  Document  (MDD).  The 
approach  taken  in  providing  this  guidance  was  to  use  the  existing  MDD  and  supplement  its 
guidance  with  that  pertinent  to  the  Government  source  selection  and  contract  process  moni¬ 
toring  usage  modes.  Two  conventions  are  used  in  the  document  to  enable  the  reader  to  distin¬ 
guish  MIG-GSS/CPM  content  from  the  MDD  text.  The  first  to  be  encountered  will  be  gray 
boxes.  These  gray  boxes  provide  explanatory  material  to  ensure  that  appropriate  interpretive 
context  is  available  to  the  reader.  The  second  convention  is  separate  pages  on  gray  paper  fol¬ 
lowing  the  individual  activity  sections  of  the  MDD.  These  pages  begin  with  the  statement, 
“When  using  the  MDD  for  Government  source  selection  and  contract  process  monitoring,  the 
following  should  be  considered.”  Three  added  appendices,  E,  F,  and  G,  are  self-explanatory. 
Every  effort  has  been  taken  to  provide  the  Government  user  with  a  single  document  contain¬ 
ing  pertinent  guidance  for  implementing  the  SCAMPI  method  in  a  source  selection  or  con- 
tract  process  monitoring  situation.  _ 


The  Method  Definition  Document  (MDD)  describes  the  Standard  CMMI®  Appraisal  Method 
for  Process  Improvement  (SCAMPIsm).  It  is  intended  to  meet  the  needs  of  different  readers. 
The  document  is  divided  into  three  major  parts,  each  with  a  different  level  of  detail,  intended 
usage,  and  primary  audience.  The  structure,  audiences,  and  suggested  use  of  each  part  of  the 
document  are  described  below. 

Document  Outline 

Part  I:  Overview 

This  part  of  the  document  provides  an  overview  of  the  method’s  context,  concepts,  and  ar¬ 
chitecture.  The  reader  is  provided  with  the  big  picture  of  the  method,  rather  than  details  about 
how  to  enact  it.  Table  1-1  shows  the  contents  of  Part  I. 


®  CMMI  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University. 
SM  CMM  Integration  and  SCAMPI  are  service  marks  of  Carnegie  Mellon  University. 
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Table  1-1:  Part  I  Contents 


Section 

Pages 

About  This  Document 

1-3  - 1-8 

Executive  Summary 

1-9 -1-13 

SCAMPI  Method  Overview 

1-15 -1-42 

Part  II:  Process  Definitions 
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tailed  activities  and  practices  associated  with  each  of  the  processes  that  compose  the 
SCAMPI  method.  Precise  listings  of  required  practices,  parameters,  and  limits  of  variation 
allowable,  as  well  as  optional  practices  and  guidance  for  enacting  the  method,  are  covered  in 
this  core  part  of  the  document.  Table  1-2  shows  the  contents  of  Part  II. 
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ment.  Rarely  will  someone  who  has  not  already  read  the  first  two  parts  read  an  appendix  of 
this  document.  The  topical  elaboration  and  reference  material  available  in  the  appendices 
help  to  provide  deeper  insight  to  an  already  knowledgeable  reader.  Table  1-3  shows  the  con¬ 
tents  of  Part  III. 
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Audiences  for  This  Document 

The  MIG-GSS/CPM  is  intended  for  authorized  Lead  Appraisers  implementing  SCAMPI  in 
Government  acquisition  environments  (source  selection  and  contract  process  monitoring). 


The  MDD  is  primarily  intended  for  SCAMPI  Lead  Appraisers  authorized  by  the  Software 
Engineering  Institute  (SEI).  It  is  expected  that  these  professionals  meet  prerequisites  for 
knowledge  and  skills  specified  by  the  SEI  Appraiser  program  (see  http://www.sei.cmu.edu/ 
for  details),  and  that  this  document  is  one  of  the  elements  of  the  infrastructure  within  which 
they  operate.  They  are  considered  the  primary  audience  for  Part  II.  Candidate  Lead  Apprais¬ 
ers  will  also  use  the  MDD  while  attending  training  to  learn  about  the  method. 

Appraisal  team  members  (under  the  leadership  of  an  authorized  SCAMPI  Lead  Appraiser) 
are  expected  to  refer  to  this  document  as  a  training  aid.  Portions  of  the  document  may  also  be 
used  as  work  aids  during  the  conduct  of  an  appraisal.  Potential  appraisal  team  members  can 
use  the  MDD  to  build  their  knowledge  base  for  future  participation  in  an  appraisal. 

Finally,  the  larger  set  of  stakeholders  for  the  conduct  of  any  given  appraisal  are  also  in  the 
targeted  audience  for  the  document,  particularly  for  Part  I.  These  stakeholders  include: 

•  appraisal  sponsors — leaders  who  sponsor  appraisals  to  meet  business  needs 

•  Process  Group  members — process  improvement  specialists  who  need  to  understand  the 
method,  and  perhaps  help  others  to  gain  familiarity 
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•  other  interested  parties  who  wish  to  have  deeper  insight  into  the  methodology  for  pur¬ 
poses  such  as  ensuring  that  they  have  an  informed  basis  for  interpreting  SCAMPI  outputs 
or  making  comparisons  among  similar  methodologies 

•  government  acquisition  personnel.  Specific  guidance  for  using  SCAMPI  in  the  Govern¬ 
ment  source  selection  and  contract  process  monitoring  environments  is  included.  Con¬ 
tracting  officers,  source  selection  evaluation  board  members,  and  other  acquisition  per¬ 
sonnel  Mil  be  interested  in  how  SCAMPI  is  used  in  the  respective  acquisition 
environments. 


How  to  Use  This  Document 

Parti 

It  is  expected  that  every  member  of  the  audience  for  this  document  will  find  value  in  Part  I. 
The  two  primary  sections  in  this  part  are  the  Executive  Summary  and  the  Method  Overview. 


The  Executive  Summary  is  intended  to  provide  high-level  information  about  what  SCAMPI 
is,  and  does  not  require  extensive  knowledge  of  appraisals.  This  portion  of  the  document  may 
be  excerpted  and  provided  to  a  more  casual  reader  or  a  stakeholder  in  need  of  general  infor¬ 
mation  to  support  their  decision  to  conduct  an  appraisal. 


The  Method  Overview  section  provides  more  comprehensive  coverage  of  SCAMPI,  and  can 
be  used  to  begin  building  a  base  of  knowledge  for  readers  who  have  need  of  more  detailed 
information.  Appraisal  sponsors  wanting  more  than  the  summary  view  described  above  will 
want  to  read  this  section.  Every  prospective  SCAMPI  team  leader  and  team  member  is  ex¬ 
pected  to  read  this  section  of  the  document,  to  ensure  that  they  have  the  big  picture  before 
study  of  the  detailed  methodology  begins. 

Part  II 

People  who  will  enact  an  appraisal  are  expected  to  read  the  second  part  of  the  document. 
Members  of  this  audience  need  to  know  how  to  enact  the  method,  not  just  what  the  method 
is.  Part  II  is  divided  into  Process  Definitions,  which  are  in  turn  divided  into  Activity  Descrip¬ 
tions.  Each  Activity  Description  delineates  Required  Practices,  Parameters  and  Limits,  Op¬ 
tional  Practices,  and  Implementation  Guidance. 


There  are  eleven  processes  contained  in  SCAMPI.  The  processes  (as  defined)  support  a  vari¬ 
ety  of  orderings  and  enactments  to  facilitate  a  variety  of  usage  modes  for  SCAMPI.  The  tem¬ 
poral  flow,  as  well  as  the  flow  of  inputs  and  outputs  among  the  processes,  is  described  in  the 
Method  Overview  section.  The  Process  Definitions  are  not  intended  to  provide  a  start-to- 
finish  view  of  SCAMPI.  Rather,  these  sections  provide  detailed  definitions  of  processes  and 
activities  that  are  invoked  according  to  the  appraisal  plan  created  by  the  appraisal  team 
leader. 
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Each  of  the  Process  Definitions  begins  with  a  three-page  overview  of  the  process.  Every 
process  is  defined  by  information  contained  in  the  elements  shown  in  Table  1-4. 


Table  1-4:  Process  Definition  Elements 


Element 

Description 

Purpose 

A  brief  summary  of  what  is  accomplished  by  enacting  the  process 

Entry  Criteria 

Conditions  that  must  be  met  before  enacting  the  process 

Inputs 

Artifacts  or  information  needed  to  enact  the  process 

Activities 

The  set  of  actions  which,  in  combination,  make  up  the  process 

Outputs 

Artifacts  and  assets  that  result  from  enacting  the  process 

Outcome 

Any  change  in  important  conditions  or  artifacts  that  results  from  en¬ 
acting  the  process 

Exit  Criteria 

Conditions  to  be  met  before  the  process  can  be  considered  complete 

Key  Points 

A  summary  of  the  most  notable  events  associated  with  the  process 

Tools  and 
Techniques 

Work  aids  commonly  used  in  enacting  the  process 

Metrics 

Useful  measures  that  support  the  process  enactment,  or  future  enactments 

Verification  and 
Validation 

Techniques  to  verify  and/or  validate  the  enactment  of  the  process 

Records 

Information  to  be  retained  for  future  use 

Tailoring 

Brief  discussion  of  key  tailoring  options  (not  an  exhaustive  list) 

Interfaces  with 

Discussion  of  how  the  process  interacts  with  other  processes  in  the 

Other  Processes 

method 

Summary  of 
Activities 

A  narrative  summary  of  the  set  of  activities 

Following  the  three  pages  of  introductory  material,  each  activity  that  is  a  part  of  the  process 
is  briefly  summarized  to  orient  the  reader  to  the  scope  of  the  activity.  Each  Activity  Descrip' 
tion  includes  the  elements  shown  in  Table  1-5. 
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Table  1-5:  Activity  Description  Elements 


Element 

Description 

Required  Practices 

A  listing  of  practices  that  must  be  implemented  to 
consider  the  enactment  a  valid  SCAMPI 

Parameters  and  Limits 

Acceptable  limits  for  things  that  are  allowed  to  vary, 
and  acceptable  limits  for  things  under  the  discretion  of 
the  appraisal  team  leader 

Optional  Practices 

Actions  that  reflect  good  practice  but  are  not  required 

Implementation  Guidance 

Narrative  description  of  advice  or  things  to  consider  in 
performing  the  activity  in  the  internal  process  im¬ 
provement  environment 

When  using  the  MDD  for  Gov¬ 
ernment  source  selection  and 
contract  process  monitoring,  the 
following  should  be  considered: 

Narrative  description  of  advice  or  things  to  consider  in 
performing  the  activity  in  the  Government  source  se¬ 
lection  (GSS)  and  contract  process  monitoring  (CPM) 
environments 

Complete  and  unambiguous  descriptions  of  the  method  processes  and  activities  are  provided 
in  this  part  of  the  document.  In  combination  with  the  training  materials  and  work  aids  that 
compose  the  CMMI  Steward’s  appraiser  program,  this  information  provides  a  firm  basis  for 
standardization  (within  reasonable  limits)  of  the  practice  of  Process  Appraisals. 

Part  III 

The  appendices  of  the  document  provide  detailed  coverage  of  special  topics  as  well  as  refer¬ 
ence  material.  Readers  knowledgeable  in  SCAMPI  are  expected  to  read  these  sections  for 
further  understanding. 


Feedback  Information 

We  are  very  interested  in  your  ideas  for  improving  this  document.  See  the  CMMI  Web  site 
for  information  on  how  to  provide  feedback:  <http://www.sei.cmu.edu/cmmi/products/ 
change-requests.html>. 


If  you  have  questions,  send  an  email  to  cmmi-comments@sei.cmu.edu. 
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Executive  Summary 


What  is  SCAMPI? 

The  Standard  CMMI  Appraisal  Method  for  Process  Improvement  (SCAMPI)  is  designed  to 
provide  benchmark  quality  ratings  relative  to  Capability  Maturity  Model  Integration 
(CMMIsm)  models.  It  is  applicable  to  a  wide  range  of  appraisal  usage  modes,  including  both 
internal  process  improvement  and  external  capability  determinations.  SCAMPI  satisfies  all  of 
the  Appraisal  Requirements  for  CMMI  (ARC)  requirements  for  a  Class  A  appraisal  method 
and  can  support  the  conduct  of  ISO/IEC  15504  assessments. 


SCAMPI  Vl.l  enables  a  sponsor  to 


•  gain  insight  into  an  organization’s  engineering  capability  by  identifying  the  strengths  and 
weaknesses  of  its  current  processes 

•  relate  these  strengths  and  weaknesses  to  the  CMMI  model 

•  prioritize  improvement  plans 

•  focus  on  improvements  (correct  weaknesses  that  generate  risks)  that  are  most  beneficial 
to  the  organization  given  its  current  level  of  oiganizational  maturity  or  process  capabili¬ 
ties 


•  derive  capability  level  ratings  as  well  as  a  maturity  level  rating 


•  identify  development/acquisition  risks  relative  to  capability/maturity  determinations 


«  I 

identify  discriminating  infom 

lation  among  multiple  suppliers  in  a  Government  soi 

ires  ":T : 

selection  environment 

:  :  L  :  :  :  :  :  : 

1 

m 

improvement  via  contract  process  monitoring  activitie 

As  a  Class  A  appraisal  method,  SCAMPI  is  an  appropriate  tool  for  benchmarking.  Sponsors 
who  want  to  compare  an  organization’s  process  improvement  achievements  with  other  orga¬ 
nizations  in  the  industry  may  have  a  maturity  level  determined  as  part  of  the  appraisal  proc¬ 
ess.  Decisions  made  on  the  basis  of  maturity  level  ratings  are  only  valid  if  the  ratings  are 
based  on  known  criteria.  Consequently,  contextual  information — organizational  scope, 
CMMI  model  scope,  appraisal  method  type,  the  identity  of  the  Lead  Appraiser  and  the 
team — are  items  for  which  criteria  and  guidance  are  provided  within  the  method  to  ensure  a 
consistent  interpretation  within  the  community.  Benchmarking  can  only  be  valid  when  there 
is  a  consistent  basis  for  establishing  the  benchmarks.  The  SEI  maintains  industry  aggregates 
for  appraisal  results.  These  data  are  reported  in  industry  maturity  profiles  gathered  from  or- 

SM  CMMI  is  a  service  mark  of  Carnegie  Mellon  University. 


CMU/SEI-2002-HB-002 


1-9 


ganizations  that  have  performed  appraisals  since  1987.  The  profile  is  based  upon  appraisal 
data  provided  by  SEI-trained  professionals,  and  is  updated  twice  annually. 

As  the  CMMI  Steward  ,  the  SEI  supports  the  SCAMPI  method  and  operates  an  authorization 
program  for  SCAMPI  Lead  Appraisers.  Additional  details  can  be  found  at 
http  ://www.  sei .  emu .  edu. 

Core  Concepts  and  Approach 

SCAMPI  VI.  1,  as  a  benchmarking  appraisal  method,  relies  upon  an  aggregation  of  evidence 
that  is  collected  via  instruments,  presentations,  documents,  and  interviews.  These  four 
sources  of  data  feed  an  “information-processing  engine”  whose  parts  are  made  up  of  a  series 
of  data  transformations.  The  appraisal  team  observes,  hears,  and  reads  information  that  is 
transformed  first  into  notes,  and  then  into  statements  of  practice  implementation  gaps  or 
strengths  (where  appropriate),  and  then  into  preliminary  findings.  These  are  validated  by  the 
organizational  unit  before  they  become  final  findings.  The  critical  concept  is  that  these  trans¬ 
formations  are  applied  to  data  reflecting  the  enacted  processes  in  the  organizational  unit  and 
the  CMMI  reference  model,  and  this  forms  the  basis  for  ratings  and  other  appraisal  results. 

Planning  is  absolutely  critical  to  the  execution  of  SCAMPI  V  1.1.  All  phase  and  process  ac¬ 
tivities  briefly  discussed  below  derive  from  a  well-articulated  plan  developed  by  the  qualified 
Lead  Appraiser,  in  concert  with  members  of  the  appraised  organization  and  the  appraisal 
sponsor. 

SCAMPI  VI. 1  Methodology 

SCAMPI  consists  of  three  phases  and  eleven  essential  processes,  as  was  shown  in  Table  1-2. 
Each  phase  is  described  in  detail  below. 

Phase  1:  Plan  and  Prepare  for  Appraisal 

The  sponsor’s  objectives  for  performing  SCAMPI  are  determined  in  phase  1,  process  1.1, 
Analyze  Requirements.  All  other  planning,  preparation,  execution,  and  reporting  of  results 
proceed  from  this  initial  activity  according  to  the  phase  and  processes  outlined.  Because  of 
the  significant  investment  and  logistical  planning  involved,  considerable  iteration  and  re¬ 
finement  of  planning  activities  should  be  expected  in  phase  1.  With  each  subsequent  phase, 
the  amount  of  iteration  will  decrease  as  data  are  collected,  analyzed,  refined,  and  translated 
into  findings  of  significance  relative  to  the  model. 

A  team  of  experienced  and  trained  personnel  performs  a  SCAMPI  over  a  period  of  time  ne¬ 
gotiated  by  the  sponsor  and  the  team  leader.  The  scope  of  the  organization  to  be  appraised 
(actual  projects  or  programs  that  will  participate),  as  well  as  the  scope  within  the  CMMI 
model  (process  areas),  must  be  defined  and  agreed  to.  The  scope  of  the  organization  and 
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model  provides  the  basis  upon  which  to  estimate  personnel  time  commitments,  logistical 
costs  (e.g.,  travel),  and  overall  costs  to  the  appraised  organization  and  to  the  sponsoring  orga¬ 
nization. 

Before  the  appraisal  begins,  members  of  the  appraised  organization  typically  prepare  objec¬ 
tive  evidence.  Objective  evidence  consists  of  qualitative  or  quantitative  information,  records, 
or  statements  of  fact  pertaining  to  the  characteristics  of  an  item  or  service  or  to  the  existence 
and  implementation  of  a  process  element.  It  is  based  on  observation,  measurement,  or  test, 
and  can  be  verified.  During  an  on-site  period,  the  appraisal  team  verifies  and  validates  the 
objective  evidence  provided  by  the  appraised  organization  to  identify  strengths  and  weak¬ 
nesses  relative  to  the  CMMI  reference  model.  The  information-processing  engine  is  thus  fu¬ 
eled  by  the  objective  evidence  already  available,  saving  the  appraisal  team  the  time  and  effort 
of  a  “discovery”  process. 

This  preparation  by  the  appraised  organization  is  critical  to  the  efficient  execution  of  a 
SCAMPI  appraisal.  Analysis  of  preliminary  objective  evidence  provided  by  the  appraised 
organization  plays  a  critical  role  in  setting  the  stage  for  the  appraisal  execution.  If  substantial 
data  are  missing  at  this  point,  subsequent  appraisal  activities  can  be  delayed  or  even  cancelled 
if  the  judgment  is  made  that  continuing  appraisal  activities  will  not  be  sufficient  to  make  up 
for  the  deficiency. 


The  collection  of  objective  evidence  by  the  appraised  organization  in  advance  of  the  appraisal 
not  only  improves  appraisal  team  efficiency,  but  also  offers  several  other  benefits  to  the  orga¬ 
nization: 

•  improved  accuracy  in  appraisal  results  delivered  by  external  appraisal  teams  (i.e.,  clear 
understanding  of  implemented  processes,  strengths,  and  weaknesses) 

•  detailed  understanding  of  how  each  process  instance  has  implemented  appraisal  reference 
model  practices,  and  the  degree  of  compliance  and  tailoring  of  organizational  standard 
processes 

•  facilities  for  monitoring  process  compliance  and  process  improvement  progress 

•  residual  appraisal  assets  that  can  be  reused  on  subsequent  appraisals,  minimizing  the  ef¬ 
fort  necessary  for  preparation 

Phase  2:  Conduct  Appraisal 

In  phase  2,  the  appraisal  team  focuses  on  collecting  data  from  the  appraised  organization  to 
judge  the  extent  to  which  the  model  is  implemented.  Integral  to  this  approach  is  the  concept 
of  “coverage,”  which  implies  (a)  the  collection  of  sufficient  data  for  each  model  component 
within  the  CMMI  reference  model  scope  selected  by  the  sponsor,  and  (b)  obtaining  a  repre¬ 
sentative  sample  of  ongoing  processes  (spanning  the  life-cycle  phases  that  the  appraised  or¬ 
ganization  is  using  in  the  development  and  delivery  of  its  products  and  services).  For  a 
benchmarking  appraisal  methodology,  this  means  collecting  data  and  information  on  all  the 
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reference  model  practices  for  each  process  instantiation  being  appraised  within  the  organiza¬ 
tional  unit.  The  data  collection  plan  developed  in  phase  1  undergoes  continuous  iteration  and 
refinement  until  sufficient  coverage  is  achieved. 

Upon  determining  that  sufficient  coverage  of  the  reference  model  and  organizational  unit  has 
been  obtained,  appraisal  findings  and  ratings  may  be  generated.  Goal  ratings  are  determined 
within  each  process  area,  which  collectively  can  be  used  to  determine  a  capability  level  rating 
for  the  individual  process  areas,  as  well  as  a  process  maturity  rating  for  the  organizational 
unit. 

Phase  III:  Report  Results 

In  phase  III,  the  appraisal  team  provides  the  findings  and  ratings  as  appraisal  results  to  the 
appraisal  sponsor  and  the  organization.  These  appraisal  results  become  part  of  the  appraisal 
record,  which  becomes  protected  data  according  to  the  desires  of  the  sponsoring  organization 
and  the  appraised  organization.  The  level  of  protection  and  the  plan  for  the  disposition  of  ap¬ 
praisal  materials  and  data  is  determined  in  phase  I  in  collaboration  with  the  sponsor.  The 
agreed-to  appraisal  record  is  also  forwarded  to  the  CMMI  Steward.  The  Steward  adds  it  to  a 
confidential  database  for  summarization  into  overall  community  maturity  and  capability  level 
profiles,  which  are  made  available  to  the  community  on  a  semiannual  basis. 

SCAMPI  Tailoring 

Successful  application  of  SCAMPI  VI.  1  relies  upon  adjusting  the  parameters  of  the  method 
to  the  needs  of  the  organization  and  to  the  objectives  and  constraints  of  the  sponsor’s  organi¬ 
zation. 

The  sponsor’s  objectives  largely  influence  tailoring  decisions.  The  CMMI  model  scope  and 
representation  (staged  or  continuous),  the  size  of  the  organizational  unit,  the  number  and  size 
of  sampled  projects,  the  size  of  the  appraisal  team,  and  the  number  of  interviews  greatly  in¬ 
fluence  things  such  as  preparation  time,  time  on  site,  and  monetary  costs,  and  so  are  also 
major  factors  when  choosing  tailoring  options.  All  tailoring  decisions  must  be  documented  in 
the  appraisal  plan. 

Tailoring  should  not  exceed  the  acceptable  limits  allowed  by  the  appraisal  method.  The 
SCAMPI  Lead  Appraiser  is  responsible  for  ensuring  that  the  requirements  of  the  method  are 
satisfied.  Tailoring  the  method  too  severely  could  result  in  the  failure  to  satisfy  method  re¬ 
quirements,  the  inability  to  obtain  sufficient  data  for  generation  of  appraisal  findings  or  rat¬ 
ings,  or  the  failure  to  meet  the  criteria  necessary  for  recognition  as  a  SCAMPI  (ARC  Class  A) 
appraisal. 
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Time  Frame  and  Personnel  Requirements 

A  nominal  time  frame  for  conducting  a  SCAMPI  appraisal  is  3  months,  including  planning, 
preparation,  and  execution.  The  follow-on  activities  implicit  with  a  full  cycle  of  appraisal  to 
re-appraisal  would  include  time  for  creating  an  action  plan  and  18  to  24  months  for  imple¬ 
mentation,  with  a  re-appraisal  occurring  in  the  latter  6  months  of  that  period.  (The  time  esti¬ 
mates  given  here  refer  to  calendar  duration  rather  than  person-months  of  effort.) 


In  the  Government  source  selection  and  contract  process  monitoring  environments,  attention 
to  schedule  and  how  the  SCAMPI  appraisal  fits  into  the  overall  acquisition  schedule  is  para¬ 
mount.  In  the  contract  process  monitoring  environment,  similar  attention  to  contract  require¬ 
ments  regarding  the  process  improvement  objectives  and  system  development  schedules  ve  il! 
be  of  pagticolar  infe^i  J  ^  V.-,  ,  ililli  EJlffi 


Personnel  needed  to  participate  in  activities  or  perform  tasks  in  a  SCAMPI  appraisal  include 
the  sponsor,  the  appraisal  team  leader,  the  Organizational  Unit  Coordinator  (OUC),  the  se¬ 
lected  participants,  and  appraisal  team  members.  Their  time  commitments  will  vary  a  great 
deal  depending  on  the  specific  parameters  of  the  appraisal  (e.g.,  organizational  scope)  and 
their  role.  Typically,  appraisal  participants  can  expect  to  spend  one  to  three  hours  each  to 
provide  objective  evidence  to  the  team  and  attend  validation  sessions.  On  the  other  extreme, 
the  OUC  may  spend  as  much  as  three  weeks  of  full  time  effort  helping  the  team  and  the  orga¬ 
nization  to  prepare  for  and  conduct  the  appraisal.  Experienced  Lead  Appraisers  will  provide 
effort  estimates  corresponding  to  the  set  of  tailoring  options  they  prefer  to  use  in  conducting  a 
SCAMPI  appraisal. 
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SCAMPI  Method  Overview 


This  section  provides  an  overview  of  the  underlying  principles  and  concepts  of  the  SCAMPI 
method.  Readers  of  the  SCAMPI  MDD  should  become  familiar  with  this  material  prior  to 
reading  the  process  descriptions  in  Part  II  of  this  document,  where  the  method  requirements 
and  tailoring  options  are  defined.  This  overview  is  primarily  targeted  at  appraisal  team  lead¬ 
ers  and  appraisal  team  members  who  will  be  performing  SCAMPI  appraisals.  Additional 
audiences  might  include  appraisal  sponsors  or  process  improvement  professionals  interested 
in  understanding  SCAMPI  features  and  the  results  that  can  be  expected. 

Method  Context 

The  SCAMPI  appraisal  method  is  used  to  identify  strengths,  weaknesses,  and  ratings  relative 
to  CMMI  reference  models.  It  incorporates  best  practices  found  successful  in  the  appraisal 
community,  and  is  based  on  the  features  of  several  legacy  appraisal  methods,  including 

•  CMM-Based  Appraisal  for  Internal  Process  Improvement  (CB A IPI)  V 1 . 1  [Dunaway 
96b]. 

•  Electronic  Industries  Alliance/Interim  Standard  (EIA/IS)  731.2  Appraisal  Method  [EIA 
98b], 

•  Software  Capability  Evaluation  (SCE)  V3.0  Method  Description  [Byrnes  96] 

•  Software  Development  Capability  Evaluation  (SDCE)  [AFMC  94] 

•  FAA  Appraisal  Method  (FAM)  [Ibrahim  99] 

SCAMPI  satisfies  the  Appraisal  Requirements  for  CMMI  (ARC)  Vl.l  [SEI  Ola]  and  is  a 
Class  A  appraisal  method. 

Method  Objectives  and  Characteristics 

The  SCAMPI  method  has  the  following  primary  objectives: 

•  Provide  a  common,  integrated  appraisal  method  capable  of  supporting  appraisals  in  the 
context  of  internal  process  improvement,  supplier  selection,  and  process  monitoring  (see 
“Modes  of  Usage”). 

•  Provide  an  efficient  appraisal  method  capable  of  being  implemented  within  reasonable 
performance  constraints  (see  “Method  Performance”). 

The  SCAMPI  method  is  also  designed  to  prioritize  and  satisfy  certain  essential  characteris¬ 
tics,  which  were  obtained  via  community  feedback  and  are  summarized  in  Table  1-6.  These 
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have  been  used  as  the  rationale  for  key  method  architecture  and  design  decisions,  which  are 
described  in  this  overview  and  throughout  the  MDD. 


Table  1-6:  Essential  Characteristics  of  the  SCAMPI  Method 


Characteristic 

Explanation 

Accuracy 

Ratings  are  truly  reflective  of  the  organization’s  maturity/capability, 
reflect  the  reference  model,  and  can  be  used  for  comparison  across  or¬ 
ganizations. 

Appraisal  results  reflect  the  strengths  and  weaknesses  of  the  appraised 
organization  (i.e.,  no  significant  strengths  and  weaknesses  are  left  un¬ 
discovered). 

Repeatability 

Ratings  and  findings  of  an  appraisal  are  likely  to  be  consistent  with 
those  of  another  independent  appraisal  conducted  under  comparable 
conditions  (i.e.,  another  appraisal  of  identical  scope  will  produce  con¬ 
sistent  results). 

Cost/Resource 

Effectiveness 

The  appraisal  method  is  efficient  in  terms  of  person-hours  spent  plan¬ 
ning,  preparing,  and  executing  an  appraisal. 

The  method  takes  account  of  the  organizational  investment  in  obtaining 
the  appraisal  results,  including  the  resources  of  the  host  organization, 
the  impact  on  appraised  projects,  and  the  appraisal  team. 

Meaningfulness 
of  Results 

Appraisal  results  are  useful  to  the  appraisal  sponsor  in  supporting  deci¬ 
sion  making.  This  may  include  application  of  the  appraisal  results  in  the 
context  of  internal  process  improvement,  supplier  selection,  or  process 
monitoring. 

ARC  Compliance 

SCAMPI  is  a  Class  A  method  and  complies  with  all  ARC  requirements. 

Modes  of  Usage 

As  used  in  the  CMMI  Product  Suite  materials,  an  appraisal  is  an  examination  of  one  or  more 
processes  by  a  trained  team  of  professionals  using  an  appraisal  reference  model  as  the  basis 
for  determining  strengths  and  weaknesses.  An  appraisal  is  typically  conducted  in  the  context 
of  process  improvement  or  capability  evaluation.  The  term  “appraisal”  is  a  generic  term  used 
throughout  the  CMMI  Product  Suite  to  describe  applications  in  these  contexts,  traditionally 
known  as  assessments  and  evaluations. 

The  basic  difference  between  an  assessment  and  an  evaluation  is  that  an  assessment  is  an  ap¬ 
praisal  that  an  organization  does  to  and  for  itself  for  the  purposes  of  process  improvement. 
Assessments  provide  internal  motivation  for  organizations  to  initiate  or  continue  process  im¬ 
provement  programs.  An  evaluation  is  an  appraisal  in  which  an  external  group  comes  into  an 
organization  and  examines  its  processes  as  input  to  a  decision  regarding  future  business. 
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Evaluations  are  typically  externally  imposed  motivation  for  organizations  to  undertake  proc¬ 
ess  improvement. 


Historically,  assessments  and  evaluations  have  been  performed  using  separate,  but  similar, 
method  descriptions,  training,  infrastructure,  and  assets.  With  version  1.1  of  the  CMMI  Prod¬ 
uct  Suite,  these  are  now  combined  into  a  single,  integrated  appraisal  methodology.  Apart 
from  this  section  of  the  MDD,  the  terms  assessment  and  evaluation  are  not  used;  the  more 
general  term  “appraisal”  is  used  throughout  to  encourage  the  recognition  of  a  single  inte¬ 
grated  method.  Concepts  from  legacy  assessment  and  evaluation  methods  are  borrowed  from 
liberally  in  the  SCAMPI  MDD,  and  representative  experts  in  these  methods  contributed  as 
core  members  of  the  ARC/MDD  product  development  team.  It  is  expected  that  users  familiar 
with  one  or  more  of  those  legacy  methods  will  be  able  to  identify  features  that  will  help  ease 
their  transition  to  the  SCAMPI  method. 


As  an  ARC  Class  A  method,  SCAMPI  is  a  benchmarking-oriented  method  suitable  for  gener¬ 
ating  ratings.  SCAMPI  appraisals  can  be  performed  in  three  modes  of  usage,  as  depicted  in 
Table  1-7.  While  many  of  the  SCAMPI  features  are  common  across  all  usage  modes  (e.g., 
identification  of  strengths,  weaknesses,  and  ratings),  there  are  differences  in  motivation  and 
intent  that  can  result  in  some  expected  method  differences  in  these  usage  modes.  The  method 
may  be  tailored  significantly  to  meet  the  business  objectives  of  the  appraisal  sponsor. 


Table  1-7:  SCAMPI  Modes  of  Usage 


Usage  Mode 

Description 

Internal  Process 
Improvement 

Organizations  use  appraisals  to  appraise  internal  processes,  generally 
to  either  baseline  their  capabihty/maturity  level(s),  to  establish  or  up¬ 
date  a  process  improvement  program,  or  to  measure  progress  in  im¬ 
plementing  such  a  program.  Applications  include  measuring  process 
improvement  progress,  conducting  process  audits,  focusing  on  specific 
domains  or  product  lines,  appraising  specific  projects,  and  preparing 
for  external  customer-led  appraisals.  In  this  manner,  SCAMPI  apprais¬ 
als  supplement  other  tools  for  implementing  process  improvement  ac¬ 
tivities. 

Supplier  Selection 

Appraisal  results  are  used  as  a  high-value  discriminator  to  select  sup¬ 
pliers.  The  results  are  used  in  characterizing  the  process-related  risk  of 
awarding  a  contract  to  a  supplier.  The  appraisal  results  are  typically 
only  one  criterion  among  many  used  to  select  suppliers.  Results  are 
often  used  as  a  baseline  in  subsequent  process  monitoring  with  the  se¬ 
lected  supplier. 
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Usage  Mode 

Description 

Process  Monitoring 

Appraisal  methods  are  also  used  in  monitoring  processes  (for  example, 
after  contract  award,  by  serving  as  input  for  an  incentive/award  fee 
decision  or  a  risk  management  plan).  The  appraisal  results  are  used  to 
help  the  sponsoring  organization  tailor  its  contract  or  process  moni¬ 
toring  efforts  by  allowing  it  to  prioritize  efforts  based  on  the  observed 
strengths  and  weaknesses  of  the  supplying  organization’s  processes. 

This  usage  mode  focuses  on  a  long-term  teaming  relationship  between 
the  sponsoring  organization  and  the  development  organization  (buyer 
and  supplier). 

See  Appendix  E  for  an  overview  and  context  for  SCAMPI  use  during  Government  source 
selection  and  contract  process  monitoring.  _ _ _  ■ 


Where  appropriate,  differences  in  the  method  requirements,  tailoring,  or  recommended  im¬ 
plementation  applicable  to  these  usage  modes  are  discussed  in  process  descriptions  and  ac¬ 
tivities  provided  in  Part  II.  These  differences  occur  most  significantly  in  the  planning  proc¬ 
esses  (e.g.,  appraisal  objectives,  sponsorship,  appraisal  planning,  selection  of  participants, 
preparation)  and  reporting  processes  (e.g.,  reporting  of  appraisal  results,  use  of  appraisal  re¬ 
sults  for  decision-making,  and  follow-on  activities). 


Note  that  the  SCAMPI  method  boundary  is  expressed  in  terms  of  enactment  of  the  appraisal 
method,  including  reporting  of  appraisal  results,  but  does  not  address  the  usage  of  those  re¬ 
sults  in  the  sponsor’s  business  context.  For  example,  the  use  of  appraisal  results  to  identify 
acquisition  risks  for  a  supplier  source  selection  is  beyond  the  scope  of  the  method.  These 
concepts  are  better  described  in  other  documentation  specific  to  those  business  contexts,  such 
as  acquisition  regulations,  standards,  and  processes. 

Method  Performance 

Performing  appraisals  efficiently  involves  minimizing  the  use  of  resources  and  the  impact  on 
appraisal  teams  and  appraised  organizations,  while  maintaining  the  essential  method  charac¬ 
teristics  that  ensure  the  high  degree  of  accuracy  required  for  an  effective  benchmarking  ap¬ 
praisal  method.  The  significantly  larger  size  of  the  CMMI  models  relative  to  legacy  source 
models  makes  this  an  even  greater  challenge. 

Method  performance  during  the  on-site  period  was  an  influential  design  driver  that  directly 
resulted  in  many  of  the  SCAMPI  features.  The  MDD  contains  many  recommendations  on 
proven,  effective  practices  that  contribute  positively  to  efficient  appraisals,  although  many  of 
these  may  not  be  strict  requirements  of  the  method.  However,  the  appraisal  method  is  only 
part  of  the  solution  for  efficient  and  cost-effective  benchmarking  appraisals  capable  of  satis¬ 
fying  all  appraisal  objectives.  Appraisal  efficiency  must  also  be  a  commitment  shared  among 
appraisal  sponsors,  appraised  organizations,  and  appraisal  teams. 
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Several  means  were  used  to  identify  targets  for  appraisal  performance  improvement,  includ¬ 
ing  analysis  of  CBA IPI  results  [Dunaway  00],  efficiency  features  from  other  appraisal  meth¬ 
ods,  CMMI  pilot  appraisals,  and  improvement  suggestions  and  best  practices  solicited  from 
the  Lead  Appraiser  community.  Ideas  were  grouped,  analyzed,  and  prioritized  for  potential 
benefit  to  method  performance  improvement.  Among  the  areas  identified  as  potential  targets 
for  performance  improvement  were  the  following: 


Table  1-8:  Potential  Targets  for  Appraisal  Performance  Improvement 


Performance 
Improvement  Topic 

Summary  Description  and  Examples 

Prework 

Thorough  appraisal  planning.  Greater  organizational  readiness. 
Pre-on-site  data  review. 

Focused  Investigation 

Focus  the  scope  of  investigation  and  follow-up  based  on  docu¬ 
mentation  review  and  questionnaire  analysis.  Continually  consoli¬ 
date  data  to  determine  progress  toward  sufficient  coverage.  Target 
further  investigation  and  interviews  on  specific  data  collection 
needs  to  optimize  effort  where  it  is  needed. 

Reuse 

Validate  results  of  prior  appraisals.  Reduce  discovery  of  earlier, 
proven  findings. 

Observations 

Reduce  time  spent  crafting  observations. 

Tailoring 

Provide  greater  clarity  on  mandatory,  suggested,  and  optional  fea¬ 
tures  of  the  method. 

Rating 

Rate  practices  (e.g.,  implemented,  partial,  not  implemented). 

Tools 

Tool  support  is  crucial  for  efficient  data  collection,  analysis,  and 
consolidation. 

Training 

Just-in-time  training.  Use  “live  data”  and  tools  for  exercises. 

Assets 

“Look-fors,”  templates,  checklists. 

Several  of  these  performance  improvement  topics  have  been  incorporated  into  the  SCAMPI 
MDD,  either  as  fundamental  method  concepts  (described  in  “Method  Concepts”  below),  or  as 
recommendations  and  implementation  options. 


Since  SCAMPI  is  suitable  for  benchmarking,  thus  requiring  high  confidence  in  ratings,  thor¬ 
oughness  is  necessary.  Organizations  for  which  (a)  generation  of  ratings  is  not  required,  (b) 
the  primary  application  is  identification  of  strengths  and  weaknesses  for  process  improve¬ 
ment,  and  (c)  efficiency  of  appraisal  resources  is  a  primary  concern  may  be  well  advised  to 
consider  alternative  appraisal  approaches.  Their  needs  may  be  satisfied  by  less  demanding 
ARC  Class  B  or  Class  C  methods.  This  is  particularly  true  for  organizations  that  are  early  in 
their  process  improvement  cycle.  Refer  to  “Requirements  for  CMMI  Appraisal  Method  Class 
Structure”  and  “Requirements  for  CMMI  Appraisal  Methods”  in  the  ARC  [SEI  Ola]  for  fur¬ 
ther  discussion  of  these  issues  and  for  guidance  in  selecting  an  appropriate  appraisal  method 
to  fit  your  business  objectives. 
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Method  Concepts 

This  section  provides  a  description  of  fundamental  concepts  employed  by  the  SCAMPI 
method.  These  concepts  are  treated  here  to  provide  readers  with  an  overall  understanding  of 
the  method  prior  to  reading  the  method  Process  Definitions  in  Part  II.  Many  of  these  concepts 
are  distributed  across  several  appraisal  method  processes  or  activities,  so  it  is  important  to 
ensure  that  a  common  understanding  is  obtained  to  recognize  the  components  of  these  con¬ 
cepts  as  they  appear  elsewhere  in  this  document. 

In  addition  to  requirements  of  the  ARC,  these  method  concepts  are  derived  from,  and  heavily 
influenced  by,  the  method  objectives,  essential  method  characteristics,  appraisal  modes  of 
usage,  and  performance  objectives  described  above. 

Method  Assumptions  and  Design  Principles 

In  addition  to  the  factors  just  mentioned,  SCAMPI  features  are  based  on  certain  method  as¬ 
sumptions  and  design  principles  related  to  the  expected  use  of  the  method.  Those  assump¬ 
tions  and  principles  are  described  below. 

SCAMPI  is  a  Class  A  benchmarking  method. 

As  an  ARC  Class  A  method,  SCAMPI  can  be  used  to  generate  ratings  as  benchmarks  to  com¬ 
pare  maturity  levels  or  capability  levels  across  organizations.  SCAMPI  is  an  integrated  ap¬ 
praisal  method  that  can  be  applied  in  the  context  of  internal  process  improvement,  supplier 
selection,  or  process  monitoring.  As  a  benchmarking  method,  the  SCAMPI  emphasis  is  on  a 
rigorous  method  capable  of  achieving  high  accuracy  and  reliability  of  appraisal  results 
through  the  collection  of  objective  evidence  from  multiple  sources. 

Goal  ratings  are  a  function  of  the  extent  to  which  the  corresponding  practices  are  present  in 
the  planned  and  implemented  processes  of  the  organization. 

In  the  CMMI  models,  there  is  a  direct  relationship  between  goals  (specific  and  generic)  and 
the  practices  (specific  and  generic)  that  contribute  toward  achievement  of  those  goals.  Spe¬ 
cific  and  generic  goals  are  required  model  components;  specific  and  generic  practices  are 
expected  model  components,  in  that  alternative  practices  could  be  implemented  that  are 
equally  effective  in  achieving  the  intent  of  the  associated  goals. 

In  the  SCAMPI  method,  a  fundamental  premise  is  that  satisfaction  of  goals  can  be  deter¬ 
mined  only  upon  detailed  investigation  of  the  extent  to  which  each  corresponding  practice  is 
implemented,  for  each  sample  instance  used  as  a  basis  for  the  appraisal  (e.g.,  each  project). 


Additional  information  on  rating  goals  is  provided  in  “Data  Collection,  Rating,  and  Report¬ 
ing”  on  page  1-26. 
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The  aggregate  of  objective  evidence  provided  is  used  as  the  basis  for  determining  practice 
implementation. 

To  make  reasonable  judgments  regarding  an  organization’s  implemented  processes  relative  to 
the  CMMI  models,  appraisal  teams  base  their  judgments  on  the  collection  of  objective  evi¬ 
dence  for  each  specific  and  generic  practice  applicable  to  process  area  goals  within  the  ap¬ 
praisal  scope. 

Appraisal  teams  compare  the  objective  evidence  collected  against  the  corresponding  practices 
in  the  reference  model.  In  making  inferences  about  the  extent  to  which  practices  are  or  are 
not  implemented,  appraisal  teams  draw  upon  the  entire  model  document  to  understand  the 
intent  of  the  model,  and  use  it  as  the  basis  for  their  decisions.  This  includes  the  required  and 
expected  model  components  (i.e.,  generic  and  specific  goals,  generic  and  specific  practices), 
as  well  as  informative  material,  such  as  model  front  matter,  introductory  text,  glossary  defi¬ 
nitions,  and  subpractices. 


Practice  implementation  at  the  organizational  unit  level  is  a  function  of  the  degree  of prac¬ 
tice  implementation  at  the  instantiation  level  (e.g.,  projects). 

Practices  described  in  the  CMMI  models  are  abstractions  that  are  realized  by  their  imple¬ 
mentation  within  projects  and  organizations.  The  context  within  which  the  practice  is  applied 
drives  the  implementation.  The  details  of  the  implementation,  as  well  as  the  context  within 
which  the  practice  is  implemented,  are  referred  to  as  the  instantiation  of  the  process,  which 
may  occur  at  the  organizational  or  project  level. 


An  organizational  unit  is  the  part  of  an  organization  that  is  the  focus  of  an  appraisal.  An  orga¬ 
nizational  unit  operates  within  a  coherent  process  context  and  a  coherent  set  of  business  ob¬ 
jectives.  It  may  consist  of  a  set  of  related  projects.  (Refer  to  the  glossary  for  a  complete  defi¬ 
nition.) 


The  extent  to  which  an  organizational  unit  has  implemented  CMMI  model  practices  can  be 
determined  only  by  considering,  in  aggregate,  the  extent  to  which  those  practices  are  imple¬ 
mented  by  instantiations  of  the  process  (i.e.,  each  sample  project  considered  within  the  ap¬ 
praisal  scope).  This,  in  turn,  necessitates  the  consideration  of  objective  evidence  for  each  in¬ 
stantiation,  for  each  model  practice  within  the  appraisal  scope. 


Appraisal  teams  are  obligated  to  seek  and  consider  objective  evidence  of  multiple  types  in 
determining  practice  implementation  and  goal  satisfaction. 

The  SCAMPI  method  is  data  oriented,  in  that  decisions  on  practice  implementation  and  rat¬ 
ing  are  made  based  upon  the  aggregate  of  objective  evidence  available  to  the  appraisal  team. 
Multiple  types  of  objective  evidence  must  be  considered;  these  are  described  in  “Objective 
Evidence  Sources”  on  page  1-23.  Artifacts  indicative  of  practice  implementation  are  a  re- 
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quirement  of  the  SCAMPI  method.  Face-to-face  interviews  are  required  to  ensure  that  the 
documentation  is  reflective  of  the  actual  organizational  process  implementation,  and  to  pre¬ 
clude  rating  judgments  being  made  solely  on  the  basis  of  artifacts.  The  SCAMPI  method  es¬ 
tablishes  minimum  requirements,  described  in  “Data  Collection,  Rating,  and  Reporting”  on 
page  1-26,  for  the  extent  to  which  objective  evidence  from  face-to-face  interviews  must  be 
collected  for  model  practices  to  corroborate  other  sources  of  objective  evidence  prior  to  rat¬ 
ing  goals. 

Verification  vs.  Discovery 

If  an  organization  has  in  place  assets,  mechanisms,  and  objective  evidence  that  substantiate 
its  implementation  of  model  practices,  it  is  in  the  organization’s  best  interest  to  share  that 
knowledge  to  ensure  that  the  appraisal  team  obtains  a  complete  and  accurate  understanding  of 
the  organization’s  implementation  of  model  practices.  Many  organizations  capture  this  un¬ 
derstanding  through  assets  such  as  traceability  and  mapping  tables  from  the  model  to  their 
organizational  processes  and  project  instantiations.  Implementation  of  the  model  within  the 
organization  may  be  further  reinforced  through  additional  mechanisms,  such  as: 

•  documentation  (e.g.,  policies,  process  descriptions,  project  plans) 

•  verification  and  oversight  activities  (e.g.,  internal  appraisals,  audits,  reviews,  status  re¬ 
ports) 

•  tools  and  resources  (e.g.,  databases,  measurement  repositories,  configuration  manage¬ 
ment  tools) 

If  assets  such  as  these,  or  indicators  of  the  existence  of  the  assets,  are  made  available  to  the 
appraisal  team,  this  leaves  the  appraisal  team  the  task  of  verifying  whether  the  objective  evi¬ 
dence  provided  is  adequate  for  substantiation  of  practice  implementation.  This  verification- 
based  approach  is  in  contrast  to  the  more  difficult,  error  prone,  and  time-consuming  task  of 
investigating  each  practice  to  discover  the  objective  evidence  needed  to  substantiate  imple¬ 
mentation.  In  a  verification-based  approach,  both  the  organizational  unit  and  the  appraisal 
team  have  a  clearer  picture  of  what  artifacts  are  available  and  what  might  still  be  needed, 
thereby  minimizing  the  amount  of  further  investigation  necessary  in  the  form  of  interviews 
and  additional  documentation  requests.  The  verification-based  approach  thus  facilitates  ap¬ 
praisals  that  are  accurate,  repeatable,  efficient,  and  that  provide  meaningful  results;  in  other 
words,  appraisals  that  satisfy  the  essential  method  characteristics  described  in  “Method  Ob¬ 
jectives  and  Characteristics”  on  page  1-15. 

Whereas  some  legacy  appraisal  methods  encouraged  organizations  to  provide  such  assets,  the 
emphasis  is  strengthened  further  in  the  SCAMPI  method,  which  is  designed  on  the  assump¬ 
tion  that  relevant  objective  evidence  is  available  for  review  in  advance  of  the  on-site  period. 
This  assumption  is  typically  discussed  with  the  appraisal  sponsor  and  his/her  representatives 
during  development  of  the  appraisal  plan.  A  key  milestone  in  the  appraisal  process  is  a  review 
prior  to  the  appraisal  on-site  period  to  determine  readiness  to  proceed  with  a  verification- 
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based  appraisal  as  planned.  If  the  appraised  organization  has  not  provided  objective  evidence 
of  sufficient  quality  and  completeness  to  enable  a  verification-based  appraisal,  the  appraisal 
plan  may  need  to  be  renegotiated  to  reflect  the  additional  effort  that  must  be  undertaken  for 
the  appraisal  team  to  search  for  and  discover  that  objective  evidence  during  the  on-site  pe¬ 
riod. 


Objective  Evidence  Sources 

The  SCAMPI  method  provides  for  the  collection  of  data  from  the  following  sources: 

•  Instruments  -  Written  information  relative  to  the  organizational  unit’s  implementation  of 
CMMI  model  practices.  This  can  include  assets  such  as  questionnaires,  surveys,  or  an  or¬ 
ganizational  mapping  of  CMMI  model  practices  to  its  corresponding  processes.  See  “In¬ 
struments  and  Tools”  on  page  1-29  for  additional  information  on  the  use  of  appraisal  in¬ 
struments. 

•  Presentations  -  Information  prepared  by  the  organization  and  delivered  visually  or  ver¬ 
bally  to  the  appraisal  team  to  describe  organizational  processes  and  implementation  of 
CMMI  model  practices.  This  typically  includes  such  mechanisms  as  orientation  or  over¬ 
view  briefings,  and  demonstrations  of  tools  or  capabilities. 

•  Documents  -  Artifacts  reflecting  the  implementation  of  one  or  more  model  practices. 
These  typically  include  organizational  policies,  procedures,  and  implementation-level  ar¬ 
tifacts.  Documents  may  be  available  in  hardcopy  or  sofitcopy,  or  may  be  accessible  via 
hyperlinks  in  a  web-based  environment. 

•  Interviews  -  Face-to-face  interaction  with  those  implementing  or  using  the  processes 
within  the  organizational  unit.  Interviews  are  typically  held  with  various  groups  or  indi¬ 
viduals,  such  as  project  leaders,  managers,  and  practitioners.  A  combination  of  formal 
and  informal  interviews  may  be  held,  using  interview  scripts  or  exploratory  questions  de¬ 
veloped  to  elicit  the  information  needed. 

Using  multiple  data-gathering  mechanisms  improves  the  depth  of  understanding  and  enables 
corroboration  of  the  data. 

Focused  Investigation 

Due  to  the  quantity  of  CMMI  model  practices  that  must  be  investigated  and  the  SCAMPI 
rules  for  collection  of  objective  evidence  to  ensure  sufficient  coverage  of  these  practices  for 
rating  (see  “Data  Collection,  Rating,  and  Reporting”  on  page  1-26),  it  is  crucial  that  appraisal 
teams  apply  efficient  techniques  for  the  collection  and  management  of  appraisal  data.  This 
focus  on  efficient  data  management  practices  is  integral  to  SCAMPI  method  concepts,  and  is 
emphasized  throughout  the  appraisal  process.  The  term  “focused  investigation”  is  used  in 
SCAMPI  to  describe  this  concept  of  optimized  investment  of  appraisal  resources.  Essentially, 
this  can  be  described  at  a  top  level  using  the  following  data  collection  and  investigation  para¬ 
digms: 
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•  Understand  what  objective  evidence  is  available,  and  how  it  contributes  toward  imple¬ 
mentation  of  model  practices  within  the  appraisal  scope. 

•  Continually  consolidate  data  to  determine  progress  toward  sufficient  coverage  of  model 
practices. 

•  Focus  appraisal  resources  by  targeting  those  areas  for  which  further  investigation  is 
needed  to  collect  additional  data  or  verify  the  set  of  objective  evidence. 

•  Avoid  unnecessary  or  duplicated  effort  that  does  not  contribute  additional  information 
toward  achievement  of  sufficient  coverage  or  toward  obtaining  significantly  greater  con¬ 
fidence  in  the  appraisal  results.  For  example,  keep  interviews  efficient  by  asking  further 
questions  only  about  practices  for  which  sufficient  data  has  not  already  been  obtained. 

These  concepts,  derived  from  the  best  practices  of  experienced  lead  appraisers,  are  primary 
mechanisms  used  to  achieve  efficient  appraisal  performance  by  emphasizing  the  placement  of 
appraisal  team  effort  where  it  is  most  needed.  This  begins  with  the  initial  collection  and 
analysis  of  objective  evidence  from  the  organizational  unit.  This  analysis  can  be  used  to  de¬ 
termine  the  adequacy  and  completeness  of  the  provided  objective  evidence,  and  to  identify 
the  extent  to  which  further  investigation  is  necessary.  The  appraisal  team’s  inventory  of  ob¬ 
jective  evidence  can  be  annotated  to  identify  practices  that  are  strongly  supported,  or  those 
that  need  further  clarification.  This  knowledge  can  be  used  as  the  basis  for  determining  find¬ 
ings  that  affect  appraisal  outcomes. 


As  the  appraisal  process  progresses,  the  appraisal  team  aggregates  and  synthesizes  additional 
objective  evidence  from  process  instantiations,  and  uses  this  to  draw  inferences  about  the 
overall  implementation  within  the  organizational  unit.  Wherever  there  are  shortcomings  in 
the  appraisal  team’s  understanding  of  the  organizational  unit’s  implementation  of  model 
practices,  data  collection  strategies  can  be  determined  to  probe  for  and  obtain  additional  in¬ 
formation.  For  example,  cases  where  the  objective  evidence  is  missing,  unclear,  or  insuffi¬ 
cient  might  be  addressed  through  additional  documentation  requests  or  by  generating  focused 
questions  for  specific  interview  participants.  By  maintaining  a  current  inventory  of  the  status 
of  the  appraisal  objective  evidence  and  prioritizing  areas  where  additional  information  is  still 
needed,  these  focused  investigation  approaches  can  be  continuously  and  iteratively  applied  to 
narrow  remaining  gaps  and  converge  on  sufficient  coverage  for  proceeding  with  rating. 


Additional  information  on  focused  investigation  and  continuous  consolidation  concepts  can 
be  found  in  Appendix  C. 

Practice  Implementation  Indicators 

The  fundamental  idea  of  Practice  Implementation  Indicators  (PIIs)  is  that  the  conduct  of  an 
activity  or  the  implementation  of  a  practice  results  in  “footprints” — evidence  that  provides  a 
basis  for  verification  of  the  activity  or  practice. 
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In  SCAMPI,  Practice  Implementation  Indicators  are  the  necessary  consequence  of  imple¬ 
menting  CMMI  model  practices.  For  example,  the  establishment  of  an  artifact,  such  as  a 
document,  is  often  an  expected  outcome  resulting  from  implementation  of  a  model  practice. 
Other  indicators  may  indirectly  substantiate  implementation  of  the  practice,  such  as  evidence 
of  a  status  meeting  or  peer  review  being  held.  Members  of  the  organizational  unit  may  affirm 
through  questionnaires  or  interviews  that  the  practice  is  implemented.  These  are  all  potential 
“footprints”  that  can  be  used  as  objective  evidence  to  verify  and  substantiate  implementation 
of  model  practices. 

SCAMPI  characterizes  PIIs  according  to  the  indicator  types  described  in  Table  1-9. 


Table  1-9:  Practice  Implementation  Indicator  Types 


Indicator  Type 

Description 

Examples 

Direct  artifacts 

The  tangible  outputs  resulting  directly 
from  implementation  of  a  specific  or  ge¬ 
neric  practice.  An  integral  part  of  verify¬ 
ing  practice  implementation.  May  be  ex¬ 
plicitly  stated  or  implied  by  the  practice 
statement  or  associated  informative  mate¬ 
rial. 

Typical  work  products  listed 
in  CMMI  model  practices 

Target  products  of  an  “Es¬ 
tablish  and  Maintain”  spe¬ 
cific  practice 

Documents,  deliverable 
products,  training  materials, 
etc. 

Indirect  artifacts 

Artifacts  that  are  a  consequence  of  per¬ 
forming  a  specific  or  generic  practice  or 
that  substantiate  its  implementation,  but 
which  are  not  the  purpose  for  which  the 
practice  is  performed.  This  indicator  type 
is  especially  useful  when  there  may  be 
doubts  about  whether  the  intent  of  the 
practice  has  been  met  (e.g.,  an  artifact 
exists  but  there  is  no  indication  of  where 
it  came  from,  who  worked  to  develop  it, 
or  how  it  is  used). 

Typical  work  products  listed 
in  CMMI  model  practices 

Meeting  minutes,  review  re¬ 
sults,  status  reports 

Performance  measures 

Affirmations 

Oral  or  written  statements  confirming  or 
supporting  implementation  of  a  specific 
or  generic  practice.  These  are  usually 
provided  by  the  implemented  of  the 
practice  and/or  internal  or  external  cus¬ 
tomers,  but  may  also  include  other 
stakeholders  (e.g.,  managers,  suppliers). 

Questionnaire  responses 

Interviews 

Presentations 

Appraisal  teams  are  obligated  to  seek  objective  evidence  of  each  of  these  types  as  a  prerequi¬ 
site  to  formulating  characterizations  of  practice  implementation.  The  indicator  types  that  will 
be  most  appropriate  to  reflect  practice  implementation  will  vary  according  to  the  context  in 
which  the  process  is  implemented,  as  well  as  the  practice  itself.  The  appraisal  team  should 
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consider  all  aspects  of  the  process  context,  such  as  project  size  and  duration,  organizational 
culture,  application  domain,  customer  market,  and  so  on,  in  determining  the  appropriateness 
and  sufficiency  of  indicators.  For  example,  the  level  of  detail  necessary  for  a  work  break- 
down  structure  will  differ  widely  for  a  1  -person,  2-week  maintenance  effort  as  opposed  to  a 
100-person,  multi-year,  mission-critical,  new  product  development. 

An  example  of  how  PIIs  for  each  of  these  types  might  be  used  in  verifying  implementation  of 
a  model  practice  is  depicted  in  Figure  1-1. 


Direct  work  product: 

Top-level  WBS,  with  revision  history 

Task  descriptions 

Work  product  descriptions 


a  top-level  work  breakdown  structure  (WBS) 


for  estimating  the  scope  of  the  project. 


Indirect  work  product:  Affirmation: 

Project  estimates  aligned  How  is  the  WBS  used? 

with  WBS  elements  How  are  estimates  generated? 

Figure  1-1:  Example  of  PH  Use 

Appraisal  teams  collect  and  organize  data  according  to  these  indicator  types.  The  SCAMPI 
method  defines  rules  and  guidelines  (described  in  “Data  Collection,  Rating,  and  Reporting” 
below)  about  the  amount  of  data  that  must  be  collected  for  each  of  these  indicator  types.  A 
combination  of  objective  evidence  according  to  these  indicator  types  is  necessary  to  corrobo¬ 
rate  multiple  sources  of  data  that  may  be  available  for  each  practice,  and  to  obtain  confidence 
in  the  accuracy  of  the  data  collected.  For  reasons  that  are  evident,  an  over-reliance  on  one 
type  of  objective  evidence  or  another  is  undesirable.  Too  much  dependence  on  artifacts  could 
result  in  the  perception  that  the  appraisal  was  a  “paper  review”  and  not  truly  indicative  of 
organizational  and/or  project  behavior.  An  over-reliance  on  affirmations  could  be  criticized  as 
not  truly  objective  or  repeatable.  Therefore,  the  SCAMPI  method  requires  a  balance  across 
these  types  of  objective  evidence. 


PP  SP1.1  Indirect  work  product: 

Minutes  of  meetings  at  which 
WBS  was  generated  or  used  to 
develop  project  estimates 


Appendix  B  contains  additional  detailed  discussion  of  PIIs  and  indicator-based  appraisals. 

Data  Collection,  Rating,  and  Reporting 

The  appraisal  team  follows  a  consensus-based,  structured  process  to  synthesize  and  transform 
information  collected  from  the  sources  described  in  “Objective  Evidence  Sources”  on  page  I- 
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23.  Data  from  these  sources  are  collected  and  considered  in  several  discrete  data-gathering 
sessions,  either  as  integrated  appraisal  team  activities  or  by  subsets  of  the  team  organized  into 
mini-teams  operating  in  parallel.  Mini-teams  are  typically  organized  around  related  process 
areas,  with  mini-team  members  assigned  by  the  appraisal  team  leader  on  the  basis  of  their 
individual  experience,  knowledge,  and  skills. 

The  SCAMPI  data  transformation  and  rating  process  is  depicted  in  Figure  1-2. 


Level  of 
Consensus 


Capability  Level  and/or 
Maturity  Level  Ratings 


Full  Team 


Goal  Satisfaction  Ratings 


Full  Team 


Practice  Implementation  Characterizations 
(organizational  unit  level) 


Full  Team 


Practice  Implementation  Characterizations 
(practice  instantiation  level) 


Mini-Team 


Figure  1-2:  SCAMPI  Rating  Process 

Team  members  review  objective  evidence  provided  by  the  organizational  unit  and  identify 
PIIs  relative  to  the  reference  model  practices.  These  PIIs  are  categorized  as  direct  artifacts, 
indirect  artifacts,  or  affirmations,  as  described  in  “Practice  Implementation  Indicators”  on 
page  1-24,  and  are  added  to  the  team’s  PII  inventory. 

Areas  of  significant  strength  or  weakness  observed  relative  to  the  implementation  of  model 
specific  or  generic  practices  are  recorded  in  written  observations.  Observations  are  generated 
primarily  for  weaknesses,  or  “gaps,”  of  the  implementation  compared  to  the  intent  of  a  model 
practice.  Observations  of  strengths  should  be  reserved  for  implemented  practices  that  are 
particularly  effective  and  are  candidates  for  inclusion  in  aggregated  findings.  “Gratuitous” 
strengths  that  simply  reflect  a  sufficient  implementation  of  a  practice  can  produce  substantial 
data  management  overhead  that  does  not  contribute  toward  generation  of  findings;  these  are 
more  effectively  captured  as  indicators  in  the  appraisal  team’s  PII  inventory.  Observations 
may  also  be  generated  for  alternative  practices,  which  are  acceptable  alternatives  to  imple¬ 
menting  one  or  more  model  practices  that  contribute  equivalently  to  the  satisfaction  of  proc¬ 
ess  area  goals. 
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Characterizing  Practices 

Verification  of  Practice  Implementation  Indicators  continues  in  this  way  at  the  instantiation 
level  until  sufficient  objective  evidence  has  been  obtained  to  characterize  the  implementation 
of  a  specific  or  generic  practice.  Sufficiency  at  the  practice  level  for  an  instantiation  is  deter¬ 
mined  when  direct  artifacts  covering  the  intent  of  the  practice  have  been  verified  for  the 
practice  and  corroborated  by  indirect  artifacts  or  affirmations.  Consensus  is  obtained  at  the 
mini-team  level  on  the  sufficiency  of  instantiation-level  practice  implementation  indicators 
and  accuracy  of  observations  of  strengths  and  weaknesses. 

Based  upon  the  practice  implementation  data  for  a  process  instantiation,  the  appraisal  team 
(or  typically  a  mini-team)  assigns  values  to  characterize  the  extent  to  which  the  CMMI  model 
practice  is  implemented.  Each  practice  is  characterized  as  Fully  Implemented  (FI),  Largely 
Implemented  (LI),  Partially  Implemented  (PI),  or  Not  Implemented  (NI).  The  intent  of  this 
characterization  is  to  effectively  summarize  the  appraisal  team’s  judgment  of  practice  imple¬ 
mentation  as  a  mechanism  to  identify  where  team  judgment  is  most  needed,  and  to  prioritize 
areas  where  further  investigation  or  corroboration  may  be  necessary.  These  characterization 
values  are  an  aid,  not  a  replacement,  for  the  observations  recorded  for  strengths  and  weak¬ 
nesses,  which  are  used  as  a  basis  for  rating  decisions. 

Upon  assigning  characterization  values  for  a  given  model  practice  for  each  instantiation,  the 
characterization  values  are  aggregated,  using  full  appraisal  team  consensus,  to  the  organiza¬ 
tional  unit  level.  Observations  reflecting  strengths  and  weaknesses  across  instantiations  are 
similarly  aggregated  to  the  organizational  unit  level,  and  form  the  basis  for  rating.  Where 
team  judgment  is  necessary  to  characterize  practice  implementation,  these  decisions  are  made 
considering  factors  such  as  the  mix  of  practice  characterizations,  the  reason  for  the  instantia¬ 
tion-level  characterizations,  and  the  severity  of  the  associated  weaknesses  (in  aggregate). 

Tracking  Progress 

The  appraisal  team  uses  focused  investigation  techniques  (see  “Focused  Investigation”  on 
page  1-23)  to  track  progress  toward  sufficient  coverage  necessary  for  rating  process  area 
goals  within  the  appraisal  scope.  Revisions  to  the  data  collection  plan  may  be  necessary  to 
ensure  that  adequate  objective  evidence  is  obtained  from  each  instantiation  (e.g.,  project)  for 
each  specific  and  generic  practice  within  the  reference  model  scope  of  the  appraisal.  If  insuf¬ 
ficient  objective  evidence  is  available,  the  data  collection  plan  may  be  revised  to  conduct  ad¬ 
ditional  data-gathering  sessions.  Focused  investigation  techniques  can  be  used  to  ensure  pro¬ 
gress  toward  sufficient  coverage  of  model  practices,  goals,  and  process  areas  within  the 
appraisal  scope. 
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Generating  Findings 

Strengths  and  weaknesses  identified  across  instantiations  are  synthesized  and  aggregated  to 
statements  of  preliminary  findings,  expressed  at  the  organizational  unit  level.  These  are  often 
organized  at  the  level  of  process  area  goals  using  common  themes.  Preliminary  findings  are 
provided  to  the  organizational  unit  for  validation;  the  mechanisms  and  timeframe  used  for 
this  may  vary  across  the  appraisal  modes  of  usage  (internal  process  improvement,  supplier 
selection,  process  monitoring).  During  this  activity,  the  appraisal  team  is  still  in  the  process  of 
collecting  data  to  ensure  that  an  accurate  understanding  of  the  organizational  process  imple¬ 
mentation  is  obtained.  Feedback  from  the  participants  in  the  appraisal  is  used  to  validate  the 
preliminary  findings,  and  may  result  in  additional  observations  or  revision  to  the  findings. 

The  appraisal  team  may  also  request  additional  data  sources  for  areas  where  their  under¬ 
standing  of  the  organization’s  implementation  of  model  practices  is  insufficient.  Final  find¬ 
ings  are  generated  based  on  the  complete,  validated  set  of  appraisal  data  (i.e.,  findings,  ag¬ 
gregated  strengths  and  weaknesses,  and  inventory  of  PIIs). 

Generating  Ratings 

Ratings  are  generated  based  on  the  set  of  validated  appraisal  data.  At  a  minimum,  ratings  are 
generated  for  each  of  the  process  area  generic  and  specific  goals  within  the  appraisal  refer¬ 
ence  model  scope.  Ratings  may  also  be  generated  for  process  areas,  capability  levels,  or  ma¬ 
turity  levels  if  desired  by  the  appraisal  sponsor.  Maturity  level  ratings  and/or  capability  level 
ratings  are  based  on  the  definitions  of  capability  levels  and  maturity  levels  in  the  CMMI 
models.  Refer  to  Process  Description  2.4,  “Generate  Appraisal  Results,”  for  additional  infor¬ 
mation  about  SCAMPI  rating  processes. 

Reporting  Results 

The  results  of  the  appraisal  are  reported  to  the  appraisal  sponsor.  For  source  selection  and 
process  monitoring  contexts,  these  results  are  also  provided  to  the  appraised  organization;  the 
mechanisms  and  timeframe  used  for  this  may  be  subject  to  acquisition  or  contractual  restric¬ 
tions.  An  appraisal  record  is  generated  and  provided  to  the  sponsor,  documenting  further  in¬ 
formation  regarding  the  appraisal. 

A  subset  of  this  data  is  provided  to  the  CMMI  Steward  for  the  purposes  of  quality  control  and 
the  collection  of  appraisal  measures  for  reporting  to  the  appraisal  community.  The  appraisal 
data  to  be  provided  is  defined  by  the  Steward  separately  from  this  document  to  allow  for 
continuous  improvement  of  appraisal  reporting  apart  from  the  CMMI  Product  Suite. 

Instruments  and  Tools 

Instruments  are  artifacts  used  in  an  appraisal  for  the  collection  and  presentation  of  data.  In¬ 
struments  are  provided  by  the  organizational  unit  to  inform  the  appraisal  team  about  the 
processes  implemented  in  the  organization  and  how  they  relate  to  the  CMMI  reference  mod- 
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els.  Instruments  can  take  various  forms,  including  questionnaires,  surveys,  site  orientation 
packets,  and  mappings  from  CMMI  practices  to  the  organizational  or  project  processes. 


The  SCAMPI  method  does  not  require  any  particular  instrument  or  presentation  format,  only 
that  an  instrument  be  used.  Instruments  can  be  used  most  effectively  if  they  provide  the  ap¬ 
praisal  team  with  an  in-depth  understanding  of  the  organizational  implementation  of  the 
model,  on  a  practice-level  basis  for  each  instantiation  to  be  investigated  in  the  appraisal.  In¬ 
struments  also  often  provide  an  opportunity  for  the  organizational  unit  to  provide  a  self¬ 
characterization  of  their  implemented  processes,  identify  applicable  substantiating  objective 
evidence,  and  specify  any  additional  comments  that  might  be  useful  in  understanding  the  im¬ 
plemented  processes.  Used  in  this  manner,  instruments  can  support  the  SCAMPI  method  em¬ 
phasis  on  verification-based  appraisals  and  minimize  the  need  for  on-site  discovery  of  objec¬ 
tive  evidence  (see  “Verification  vs.  Discovery”  on  page  1-22),  thus  helping  to  facilitate 
efficient  appraisal  performance. 


As  described  in  “Practice  Implementation  Indicators”  on  page  1-24,  the  SCAMPI  method 
emphasizes  the  use  of  PIIs.  Organizations  may  provide  as  input  to  the  appraisal  a  PII  data¬ 
base  (PIIDB),  with  a  mapping  of  model  practices  to  corresponding  processes  and  objective 
evidence  that  can  be  used  to  verify  practice  implementation.  It  is  anticipated  that  many  orga¬ 
nizations  will  have  existing  assets  in  place  that  reflect  their  process  implementation  and  map¬ 
ping  to  CMMI  model  practices.  These  instruments  can  be  used  as  a  source  of  appraisal  data 
in  much  the  same  way  as  a  PIIDB.  The  collection  of  these  model  mappings  and  indicators 
can  be  a  valuable  resource  for  process  improvement  at  the  organization  and  project  levels, 
and  a  rich  source  of  data  for  process  appraisals  using  a  variety  of  Class  A,  B,  and  C  appraisal 
methods. 

It  is  recommended  that  a  member  of  the  appraisal  team  facilitate  the  entry  of  data  into  in¬ 
struments  where  feasible,  to  ensure  that  appropriate  data  are  obtained.  This  can  help  the  ap¬ 
praised  organization  clarify  or  interpret  the  intent  of  the  model  practices,  understand  what 
data  are  expected,  and  focus  the  responses.  The  entry  of  either  too  much  or  too  little  data  into 
instruments  can  be  problematic  for  both  the  appraisal  team  and  the  appraised  organization 
and  result  in  inefficient  use  of  resources. 

Effective  management  of  appraisal  data  is  a  significant  challenge  that  can  be  simplified  with 
the  use  of  automated  tools.  The  CMMI  Steward  provides  a  rudimentary  toolkit  to  Lead  Ap¬ 
praisers  that  can  be  used  to  collect  practice -level  questionnaire  data;  characterize,  consoli¬ 
date,  and  summarize  responses;  and  record  observations  based  on  these  responses  where  ap¬ 
propriate.  Several  vendor  tools  are  also  available  in  the  marketplace.  The  choice  of  tools  is 
largely  one  of  personal  preference;  some  experienced  appraisers  prefer  manual  techniques, 
such  as  wall  charts,  to  record  observations  and  findings. 
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Effective  Team  Practices 


Appraisal  team  dynamics  and  effective  group  techniques  contribute  significantly  to  the  ability 
to  conduct  SCAMPI  appraisals.  The  appraisal  team  leader  can  help  focus  team  activities  so 
that  effort  is  spent  wisely  toward  achievement  of  method  requirements  and  appraisal  objec¬ 
tives.  “Method  Performance”  on  page  1-18  contains  several  efficiency  ideas  identified  by  the 
appraisal  community  as  potential  areas  for  improvement.  SCAMPI  features  encourage  effec¬ 
tive  team  practices  that,  with  the  support  of  the  appraisal  team  leader,  can  address  some  of 
these  issues.  This  includes  areas  such  as: 

•  Verification-based  approach  -  Verification  of  PIIs  provided  as  objective  evidence  by  the 
organization  in  advance  of  the  appraisal  is  emphasized  to  reduce  the  extent  of  data  that 
must  be  obtained  through  discovery  techniques  during  the  on-site  period.  Even  in  this 
case,  it  is  recommended  that  the  entry  of  PII  data  by  the  organizational  unit  be  facilitated 
to  ensure  that  an  appropriate  and  useful  set  of  objective  evidence  is  available;  too  much 
data  that  is  not  useful  is  just  as  great  a  problem  as  too  little  data.  (See  “Verification  vs. 
Discovery”  on  page  1-22.) 

•  Reduced  crafting  of  observations  -  In  an  indicator-based  appraisal,  greater  emphasis  is 
placed  on  verification  of  PIIs,  and  there  is  less  need  overall  for  crafting  notes  and  obser¬ 
vations.  Observations  need  not  be  generated  simply  to  acknowledge  satisfactory  imple¬ 
mentations  or  existence  of  artifacts,  but  can  focus  more  on  identifying  weaknesses  or  sig¬ 
nificant  strengths  that  can  be  expected  to  be  included  in  the  findings. 

•  Consensus  -  Mini-teams  are  given  the  authority  to  reach  consensus  on  practice  imple¬ 
mentation  at  the  instantiation  level;  full  team  consensus  is  required  for  aggregation  to  the 
organizational  unit  level.  (See  Figure  2.2.6-1.)  The  characterization  of  practice  imple¬ 
mentation  (FI,  LI,  PI,  NI;  see  “Data  Collection,  Rating,  and  Reporting”  on  page  1-26)  can 
also  help  facilitate  consensus  on  whether  implementations  satisfy  model  intent,  either  at 
the  instantiation  or  organizational  unit  level.  The  consensus,  discussion,  and  decision¬ 
making  processes  used  by  the  appraisal  team  can  be  significant  sources  of  inefficiency  if 
not  monitored  closely. 

•  Corroboration  -  Corroboration  is  built  into  the  method  through  requirements  for  multiple 
types  of  objective  evidence  (direct  artifacts,  indirect  artifacts,  affirmations).  (See  Section 
2.2.5.) 

•  Effective  data  management  -  The  SCAMPI  method  provides  ways  to  collect,  organize, 
and  manage  appraisal  data  efficiently,  and  to  facilitate  the  team  decisions  that  must  be 
made  based  on  the  set  of  objective  evidence.  The  focused  investigation  techniques  de¬ 
scribed  in  Section  2.2.4  can  help  keep  the  team  oriented  on  what  objective  evidence  has 
been  collected,  what  remains  to  be  collected,  and  how  it  will  be  collected.  This  can  be 
greatly  enhanced  through  the  use  of  automated  support  tools.  A  thorough  understanding 
of  progress  toward  sufficiency  of  coverage  can  help  focus  data  collection.  Interviews,  in 
particular,  can  be  shortened  by  focusing  on  specific  data  collection  needs. 

Several  additional  effective  team  practices  are  targeted  toward  specific  subsets  of  the  ap¬ 
praisal,  and  are  included  as  suggested  implementation  or  tailoring  options  within  individual 
process  descriptions  in  Part  II. 


CMU/SEI-2002-HB-002 


1-31 


Method  Description 

This  section  provides  an  overview  of  the  SCAMPI  method  architecture,  including  appraisal 
phases,  processes,  and  activities.  These  descriptions  are  high-level  abstractions  of  the  process 
descriptions  contained  in  Part  II  of  this  document. 


A  summary  of  the  SCAMPI  method  processes  and  activities  for  each  of  the  three  appraisal 
phases  is  contained  in  Tables  I- 10  through  1-12. 

The  interactions  between  the  processes  are  depicted  in  the  process  flow  diagrams  in  Figures 
1-3  through  1-5.  These  diagrams  show  the  work  products  that  are  inputs  and  outputs  at  the 
process  level  for  accomplishing  the  purpose  of  the  appraisal.  Additional  analysis  was  done  to 
ensure  that  the  activities  within  each  process  use  and  provide  the  inputs  and  outputs  of  the 
process.  However,  that  detailed  analysis  is  not  presented  here. 


The  process  flows  generally  show  summarized  and  completed  products.  For  instance,  the  ap¬ 
praisal  input  generated  by  Analyze  Requirements  initially  is  provided  to  the  Develop  Ap¬ 
praisal  Plan  process  with  some  elements  missing  that  are  generated  in  other  processes.  These 
flow  back  to  Analyze  Requirements  in  the  appraisal  plan.  The  final  appraisal  input  as  coordi¬ 
nated  with  the  sponsor  is  then  produced  as  a  completed  product.  Additional  administrative 
and  support  products,  such  as  appraisal  checklists,  will  be  produced  but  are  not  included  in 
these  diagrams. 

The  time  sequences  of  appraisals  are  also  shown  in  the  process  flow  diagrams.  Figures  1-6 
and  1-7  show  nominal  schedules  for  conduct  of  appraisals  in  both  assessment  and  evaluation 
modes.  There  are  several  differences  between  and  tailoring  options  within  each  of  these 
schedules.  They  are  examples  and  are  not  intended  to  be  requirements. 


For  assessments,  the  preparation  of  participants  (“Prep”  in  the  diagram)  can  be  at  any  time 
between  the  identification  of  the  participants  and  the  administration  of  the  instruments.  If  a 
set  of  PIIs  assembled  by  the  organization  for  previous  appraisals  is  chosen  as  the  instrument, 
this  would  be  at  the  beginning  of  data  collection.  If  an  additional  completion  of  instruments 
beyond  those  provided  in  the  initial  objective  evidence  is  not  required,  this  preparation  could 
be  delayed  to  just  prior  to  interviews.  Other  options  include  but  are  not  limited  to  the  timing 
of  team  selection,  number  and  timing  of  readiness  reviews,  and  sequence  of  artifact  and  in¬ 
terview  activities. 

For  evaluations,  there  are  several  differences  in  sequence  from  assessments.  For  instance,  the 
analysis  of  initial  objective  evidence  occurs  after  the  organizations  have  responded  to  the  re¬ 
quest  for  data,  which  in  turn  follows  the  completion  of  the  Data  Collection  Plan.  Another  dif¬ 
ference  is  the  delay  of  the  delivery  of  the  appraisal  results  until  after  all  organizations  have 


1-32 


CMU/SEI-2002-HB-002 


been  appraised.  An  example  of  timing  options  is  that  the  identification  of  the  appraisal  team 
leader  may  be  delayed  until  near  the  end  of  the  planning  activities.  This  ordering  of  events 
must  be  accommodated  by  the  plan;  for  example,  completion  of  the  Analyze  Requirements 
and  Develop  Appraisal  Plan  processes  must  be  rescheduled  to  allow  the  appraisal  team  leader 
to  approve  the  appraisal  input  and  the  appraisal  plan. 
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Table  1-10:  SCAMPI  Phase  Summary:  Plan  and  Prepare  for  Appraisal 
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Table  1-11:  SCAMPI  Phase  Summary:  Conduct  Appraisal 
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Prepared  Team 


Figure  1-4:  Process  Flows,  Conduct  Appraisal  Processes 
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Nominal  Assessment  Schedule  View 
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Nominal  Evaluation  Schedule  View 
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Part  II:  Process  Definitions 
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1 .1  Analyze  Requirements 


Purpose 

Entry  Criteria 

inputs 

Activities 

Outputs 

Outcome 

Exit  Criteria 


Understand  the  business  needs  of  the  organization  for  which  the  appraisal  is 
being  requested.  The  appraisal  team  leader  will  collect  information  and  help 
the  appraisal  sponsor  match  appraisal  objectives  with  their  business 
objectives. 


•  An  appraisal  sponsor  has  decided  that  a  SCAMPI  appraisal  should  be 
performed. 

•  People  who  can  provide  statements  of  requirements  for  the  appraisal  are 
available. 


•  Sponsor 

•  Initial  requirements  and  constraints 

•  Process-related  legacy  information 


1.1.1  Determine  Appraisal  Objectives 

1.1.2  Determine  Appraisal  Constraints 

1.1.3  Determine  Appraisal  Scope 

1.1.4  Determine  Outputs 

1.1.5  Obtain  Commitment  to  Appraisal  Input 


The  appraisal  input 


The  decision  to  proceed  with  the  appraisal  based  on  a  shared  understanding  of 
the  appraisal  objectives,  constraints,  outputs,  and  scope. 


•  Initial  contact  between  the  appraisal  sponsor  and  authorized  SCAMPI 
Lead  Appraiser  has  occurred. 

•  The  Lead  Appraiser  has  been  given  access  to  members  of  the  sponsoring 
organization. 

•  The  appraisal  input  has  been  approved  by  the  appraisal  sponsor  and 
placed  under  change  management. 


Continued  on  next  page 
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1 .1  Analyze  Requirements  (continued) 


Key  Points 


Tools  and 
Techniques 


Metrics 


Verification  and 
Validation 

Records 

Tailoring 


At  this  early  stage  in  the  process,  gathering  information  that  supports  good 
planning  is  most  important.  Often,  the  appraisal  team  leader  must  educate 
members  of  the  sponsor’s  organization  in  the  purpose  and  role  of  appraisals. 


Collaborative  consultation  between  the  appraisal  team  leader  and  the  appraisal 
sponsor  is  important  in  this  activity.  The  appraisal  team  leader  may  be  able  to 
simply  interview  the  sponsor  to  get  the  needed  information  and  reach 
agreements.  In  some  settings,  a  series  of  meetings  with  different  stakeholders 
may  be  needed  to  elicit  and  build  consensus  on  the  business  needs  that  can  be 
met  through  a  SCAMPI  appraisal. 

Understanding  the  history  of  appraisals  in  the  organization,  especially  the 
organizational  and  model  scope  of  past  appraisals,  is  important  for 
understanding  the  requirements  for  the  appraisal  under  consideration.  The 
choices  sponsors  make  about  appraisal  scope  are  often  tied  to  their 
(sometimes-unstated)  priorities  for  process  improvement. 


A  number  of  metrics  support  the  appraisal  team  leader’s  monitoring  of  this 
work: 

•  calendar  time  between  initial  contact  and  finalization  of  requirements 

•  effort  expended  to  gather  and  analyze  requirements 

•  number  of  meetings  with  representatives  of  the  sponsoring  and/or 
appraised  organization 


The  exit  criterion  for  this  activity  is  the  formal  approval  of  the  appraisal  input 
and  its  placement  under  change  management. 

Review  of  the  documented  agreements  resulting  from  the  work  of  this  set  of 
activities  will  serve  to  validate  the  requirements,  which  feed  into  appraisal 
planning. 


The  appraisal  input 


The  experience  of  the  sponsor  with  process  appraisals  will  drive  tailoring 
choices  for  this  process. 

•  A  relatively  inexperienced  appraisal  sponsor  will  need  a  great  deal  of 
information  and  collaborative  consultation  to  provide  meaningful  and 
complete  requirements  for  the  appraisal. 

•  Experienced  sponsors  may  have  overly  aggressive  requirements. 


Continued  on  next  page 
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1.1  Analyze  Requirements  (continued) 


Interfaces 
with  Other 
Processes 


This  process  is  a  foundation  for  the  success  or  failure  of  the  entire  appraisal;  it 
is  at  this  point  in  the  appraisal  that  the  most  leverage  exists  for  avoiding 
problems  and  issues  downstream.  Gathering  and  understanding  the 
requirements  for  the  conduct  of  a  SCAMPI  appraisal  is  vital  to  making 
appropriate  decisions  and  providing  value  to  the  sponsor.  Many  examples  of 
problems  encountered  during  appraisals  can  be  traced  to  shortcomings  in  the 
conduct  of  this  process.  The  extent  to  which  the  activities  described  here  are 
distinct  from  the  activities  described  in  the  next  process.  Develop  Appraisal 
Plan,  will  depend  on  the  strategy  and  preferences  of  both  the  appraisal  team 
leader  and  the  appraisal  sponsor. 


Summary  of  The  objectives  that  motivate  the  conduct  of  an  appraisal  must  be  well 
Activities  understood  so  that  appropriate  participants,  tailoring  decisions,  and  appraisal 

outputs  can  be  selected.  The  constraints  that  shape  the  appraisal  enactment,  in 
light  of  the  objectives,  may  limit  achievement  of  the  desired  result  if  they  are 
not  adequately  understood  and  negotiated.  A  clear  agreement  regarding 
appraisal  outputs  and  their  intended  usage  will  help  maintain  the  sponsorship 
needed  for  conducting  the  appraisal  and  acting  on  the  results.  Establishing 
agreement  on  these  objectives,  constraints,  outputs,  and  intended  usage  forms 
the  basis  for  a  commitment  to  the  plan  for  conducting  the  appraisal. 
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1.1  Analyze  Requirements 


GSS  and  CPM  Implementation  Guidance 

When  using  the  MDD  for  Government  source  selection  and  contract  process  monitoring,  the 

following  should  be  considered: 

Guidance  for  The  process  of  analyzing  requirements  should  be  started  early  during  the 

GSS  and  CPM  activity  of  defining  the  acquisition  strategy  for  the  procurement.  For  both  GSS 

and  CPM  uses,  the  detailed  planning  performed -in  process  1,2,  Develop 
Appraisal  Plan,  must  result  in  specific  information  about  the  use  of  SCAMPI 
to  be  included  in  the  source  selection  planning  documents  and  specific 
wording  to  be  included  in  the  RFP.  In  addition,  the  Federal  Business 
Opportunity  (FBO)l  Commerce  Business  Daily  ( CBD )  announcement  of  the 
procurement  must  include  notification  of  the  intent  to  perform  a  SCAMPI 
appraisal  in  the  case  where  a  SCAMPI  appraisal  is  to  be  used  in  GSS.  The 
process  of  analyzing  requirements  should  begin  early  enough  to  provide  the 
appraisal  input  to  the  detailed  planning  process  in  time  to  support  preparation 
of  these  products.  Experience  has  shown  that  the  process  of  analyzing 
requirements  takes  approximately  four  to  five  weeks.  Therefore,  it  is 
recommended  that  this  process  be  started  at  least  four  weeks  before  the 
issuance  of  any  FBOICBD  announcement  that  includes  the  intent  to  perform 
an  appraisal  or  the  beginning  of  the  RFP  preparation,  whichever  comes  first. 
Since  the  sponsor’s  decision  to  perform  a  SCAMPI  appraisal  is  an  entry 
condition  for  this  process,  the  decision  to  perform  a  SCAMPI  appraisal  must 
have  been  made  before  that  time. 

Large  programs  are  frequently  procured  using  a  multi-phased  approach 
involving  a  sequence  of  contracts  (e.g..  Concept  and  Technology 
Development  (CTD),  System  Development  and  Demonstration  (SDD), 
Production  and  Deployment  (PD),  and  Operations  and  Support  phases  for 
DoD  programs).  In  this  type  of  approach,  earlier  phases  frequently  involve 
multiple  parallel  contracts,  with  the  number  of  contracts  being  reduced  in  each 
phase  until  a  single  supplier  is  selected  to  complete  the  system  development. 
While  most  of  the  development  is  usually  performed  during  SDD,  significant 
systems  and  software  engineering  critical  to  the  future  direction  of  the 
program  may  be  performed  in  the  CTD  phase.  Therefore,  program  objectives 
may  best  be  met  by  performing  one  or  more  SCAMPI  appraisals  for  GSS 
and/or  CPM  early  in  the  program  life  cycle  rather  than  waiting  until  SDD.  For 
this  type  of  program,  the  process  of  analyzing  requirements  for  the  SCAMPI 
appraisal(s)  should  begin  shortly  after  the  program  is  initiated  in  order  to 
determine  the  appraisal  strategy  best  suited  to  meeting  program  objectives. 


Continued  on  next  page 
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1.1  Analyze  Requirements 


GSS  and  CPM  Implementation  Guidance  (continued) 

Guidance  for  The  amount  of  time  that  an  external  Lead  Appraiser  can  spend  with  the 

GSS  and  CPM,  sponsoring  organization  during  the  requirement  analysis  and  planning 

cont.  processes  may  be  quite  limited.  Therefore,  it  will  usually  be  necessary  to  have 

an  appraisal  point  of  contact  (POC)  within  the  sponsoring  acquisition 
organization  to  assist  the  Lead  Appraiser  in  the  preparation  for  the  appraisal. 
Additionally,  the  process  of  analyzing  requirements  generally  begins  before 
the  Lead  Appraiser  is  identified.  Furthermore,  some  sponsoring  acquisition 
organizations  will  not  necessarily  have  qualified  Lead  Appraisers  available 
and  will  need  to  obtain  the  services  of  a  Lead  Appraiser  from  outside  the 
sponsoring  organization.  As  a  result,  the  appraisal  POC  must  have  expertise  in 
how  acquisitions  are  performed  in  the  sponsoring  organization  and  must  be 
knowledgeable  about  the  CMMI  models  and  SCAMPI  method.  The  appraisal 
POC  will  be  responsible  for  performing  the  activities  of  the  requirements 
analysis  and  planning  processes  in  close  coordination  with  the  Lead 
Appraiser. 

Although  the  identification  of  the  Lead  Appraiser  is  an  exit  criterion  for 
process  1.1,  Analyze  Requirements,  the  Lead  Appraiser  should  be  identified 
as  early  as  possible  during  this  process.  Early  participation  by  the  Lead 
Appraiser  will  help  ensure  that  the  requirements  are  sufficiently  well  defined 
to  form  a  good  basis  for  the  detailed  planning  to  be  performed  in  the  next 
process  and  to  ensure  that  the  results  of  the  requirements  analysis  process  are 
thoroughly  understood  by  the  sponsoring  acquisition  organization. 

Note  that  it  is  very  important  for  the  sponsoring  acquisition  organization  to 
understand  the  purpose  and  role  of  a  SCAMPI  appraisal  in  GSS  and/or  CPM 
and  what  a  SCAMPI  appraisal  entails.  It  is  not  enough  for  only  the  appraisal 
sponsor  and  appraisal  POC  to  understand  this.  The  program  manager. 
Contracting  Officer,  and  personnel  responsible  for  the  acquisition  strategy  and 
source  selection  must  all  have  a  basic  understanding  of  what  is  involved.  The 
Lead  Appraiser  and  appraisal  POC  must  take  on  the  job  of  educating  the 
sponsoring  acquisition  organization. 

The  results  of  the  requirements  analysis  process  may  lead  to  appraisal  input 
that  precludes  an  ARC  Class  A  appraisal  from  being  performed.  In  such  a 
case,  it  may  be  possible  to  define  an  ARC  Class  B  or  Class  C  appraisal  that 
meets  the  requirements.  In  any  case,  process  1.1,  Analyze  Requirements, 
should  be  performed  in  its  entirety,  no  matter  what  the  eventual  class  of  the 
appraisal.  Thorough  analysis  of  requirements  is  essential  to  any  successful 
appraisal. 
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1 .1  Analyze  Requirements 


GSS  and  CPM  Implementation  Guidance  (continued) 

Guidance  for  Activities  1.1.1  through  1.1.4  of  this  process  are  performed  concurrently  and 
GSS  and  CPM,  iteratively  until  an  agreed-to  set  of  appraisal  input  is  obtained  (activity  1.1.5). 
cont.  In  addition,  process  1.1  may  be  performed  iteratively  and  concurrently  with 

processes  1.2  through  1.5  as  the  preparation  for  the  appraisal  proceeds.  It  is 
very  important  that  the  activities  in  these  processes  be  integrated  and 
consistent  with  the  other  procurement  activities  that  are  being  done 
simultaneously  (e.g.,  definition  of  the  acquisition  strategy,  preparation  of  the 
RFP  and  source  selection  planning  documents)  to  guarantee  that  the  SCAMPI 
appraisal  is  properly  integrated  into  the  procurement.  It  is  also  vital  that  these 
activities  address  the  acquirer’s  long-term  strategy  for  mitigating  process  risks 
for  the  program,  including 

•  the  process  improvement  activities  that  will  be  expected  of  offerors/ 
development  contractors  to  address/mitigate  any  process  issues  identified 
by  the  appraisals 

•  the  acquirer’s  strategy  to  (a)  incentivize  the  contractor’s  mitigation  or 
avoidance  of  process  related  risks  and/or  (b)  reappraise  the  subject 
organization  to  identify  trends  in  the  program’s  process  maturity 
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1 .1 .1  Determine  Appraisal  Objectives 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


The  business  needs  for  process  improvement  drive  the  requirements  for  the 
conduct  of  any  given  appraisal,  and  generally  include  one  or  more  of  three 
closely  related  factors: 

•  reducing  costs 

•  improving  quality 

•  decreasing  time  to  market 

The  fundamental  premise  of  process  improvement  is  that  organizational 
processes  significantly  impact  these  factors. 

A  fair  and  objective  characterization  of  the  processes  in  use  in  the 
organization(s)  is  the  essential  reason  for  conducting  an  appraisal.  In  addition 
to  this  motivation,  a  sponsor’s  desire  to  conduct  an  appraisal  could  be  driven 
by  one  or  more  of  the  following  business-related  objectives: 

•  Document  a  credible  benchmark  that  reflects  successful  process 
improvement. 

•  Evaluate  areas  of  potential  risk  that  may  affect  the  performance  of  the 
organization. 

•  Involve  members  of  the  appraised  organization  in  improving  the 
performance  of  the  process. 

•  Support  specific  decisions  related  to  the  direction  of  a  new  or  existing 
improvement  program. 

•  Motivate  a  supplier  to  focus  on  process  issues  that  affect  contract 
performance. 


•  Identify  sponsor  and  relevant  stakeholders,  and  establish  communication. 

•  Document  business  and  appraisal  objectives. 

•  Ensure  the  alignment  of  appraisal  objectives  to  business  objectives. 

•  Determine  and  document  the  appraisal  usage  mode  (Internal  Process 
Improvement,  Supplier  Selection,  Process  Monitoring). 


At  least  one  communication  between  the  appraisal  team  leader  and  sponsor  is 
required.  (Some  usage  modes  may  limit  this  significantly;  others  may  require 
more  than  one  interaction.) 


Continued  on  next  page 
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1 .1 .1  Determine  Appraisal  Objectives  (continued) 


Optional  None. 

Practices 


Implementation  Organizations  with  experience  in  the  use  of  appraisals  may  have  a  clear  set  of 
Guidance  appraisal  objectives  identified  in  advance  of  contacting  a  Lead  Appraiser.  This 

provides  the  Lead  Appraiser  with  a  starting  point,  but  does  not  permit  him  or 
her  to  “skip”  this  activity. 

In  some  cases  the  usage  mode  will  be  self-evident;  however,  there  may  bd 
instances  where  the  appraisal  sponsor  either  may  not  be  sure  or  may  have 
made  an  assumption  that  is  not  founded  on  fact.  The  appraisal  team  leader  is 
responsible  for  ensuring  that  the  best  choice  of  usage  mode  is  made  consistent 
with  the  sponsor’s  input  and  direction. 

Also  note  that  the  roles  of  appraisal  sponsor  and  senior  site  manager  may  be 
played  by  the  same  person  or  by  two  individuals,  depending  on  the  usage 
mode. 

Depending  on  the  structure  of  the  appraised  organization,  as  well  as  the  usage 
mode,  it  is  often  important  to  distinguish  the  role  of  senior  site  manager  from 
that  of  appraisal  sponsor.  For  some  appraisals,  these  two  roles  are 
encompassed  in  the  duties  of  a  single  person.  For  other  appraisals,  these  two 
roles  may  represent  two  people  working  many  time  zones  away  from  each 
other. 
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1 .1 .1  Determine  Appraisal  Objectives 


GSS  and  CPM  implementation  Guidance 

When  using  the  MDD  for  Government  source  selection  and  contract  process  monitoring,  the 
following  should  be  considered: 

Guidance  for  Use  of  a  SCAMPI  appraisal  for  GSS  or  CPM  is  usually  motivated  by  one  or 
GSS  and  CPM  more  of  the  following  four  fundamental  business  objectives  for  the  program: 

•  Achieve  program  cost  objectives. 

•  Achieve  program  schedule  objectives. 

•  Achieve  program  quality  objectives. 

•  Achieve  program  performance  objectives. 

Specific  objectives  for  an  GSS  SCAMPI  appraisal  may  include  one  or  more  of 

the  following: 

•  Determine  discriminators  between  offerors  in  the  area  of  process 
capability  (e.g.,  SW,  SE,  BPPD,  SS)  to  support  source  selection. 

•  Identify  risks  in  process  capability  that  may  affect  contract  performance 
and/or  that  should  be  managed  by  the  acquisition  organization  after 
contract  award. 

•  Obtain  contractual  commitment  to  use  mature  processes. 

•  Satisfy  policy  or  regulations  applicable  to  the  sponsoring  acquisition 
organization. 

Specific  objectives  for  CPM  SCAMPI  appraisals  may  include  one  or  more  of 

the  following: 

•  Motivate  a  supplier  to  focus  on  process  issues  that  affect  contract 
performance  (e.g.,  through  use  of  award  and/or  incentive  fees  connected 
to  process  improvement). 

•  Involve  members  of  the  supplier  team  in  improving  process  performance 
on  the  program. 

•  Identify  and  manage  risks  in  process  capability  that  may  affect  contract 
performance. 

•  Motivate  compliance  with  contractual  commitment  to  process 
performance. 
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1 .1 .1  Determine  Appraisal  Objectives 


GSS  and  CPM  Implementation  Guidance  (continued) 

Guidance  for  Since  SCAMPI  appraisals  may  be  used  for  both  GSS  and  CPM  on  the  same 
GSS  and  CPM,  contract,  both  types  of  appraisal  objectives  may  apply  to  a  single  program. 

Note  that  for  GSS  and  CPM  uses  of  a  SCAMPI  appraisal,  the  appraisal 
sponsor  and  senior  site  manager  are  always  different  people,  with  the  former 
being  a  member  of  the  sponsoring  acquisition  organization  and  the  latter  being 
a  member  of  the  organization  undergoing  the  appraisal. 

Determination  of  appraisal  objectives  needs  to  also  address  follow-on 
activities  expected  as  a  result  of  the  appraisal: 

•  program  risk-mitigation  activities  to  address  process  weaknesses  post 
contract  award 

•  action  plans  established  in  support  of  process  improvement  activities 

•  frequency/timing  of  reappraisals  to  determine  the  program’s  effectiveness 
in  addressing  process  issues  throughout  the  life  cycle 

♦ 
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1 .1 .2  Determine  Appraisal  Constraints 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


The  constraints  within  which  the  appraisal  must  be  conducted  are  determined 
based  on  a  dialog  between  the  appraisal  team  leader  and  the  appraisal  sponsor 
and/or  senior  site  manager.  This  typically  is  an  iterative  process  in  which  the 
preferences  of  the  appraisal  sponsor,  the  limits  of  the  method,  and  the 
consequent  resource  requirements  are  balanced  against  each  other  to  arrive  at 
an  optimal  set  of  appraisal  input  parameters. 


•  Establish  high-level  cost  and  schedule  constraints. 

•  Determine  which  process  areas  (PAs)  and  organizational  entities  axe  to  be 
included. 

•  Determine  minimum,  maximum,  or  specific  sample  size  or  coverage  that 
is  desired  for  the  appraisal. 

•  Negotiate  constraints  and  objectives  with  stakeholders  to  ensure 
feasibility. 

•  Document  negotiated  constraints  to  be  met. 


At  least  one  communication  between  the  appraisal  team  leader  and  sponsor  is 
required.  (Some  usage  modes  may  limit  this  significantly;  others  may  require 
more  than  one  interaction.) 

Constraints  on  cost  and  schedule  identified  during  this  early  stage  of  the 
appraisal  are  expected  to  be  high-level,  and  not  detailed  estimates.  They  may 
take  the  form  of  statements  such  as  “We  need  this  done  in  Q4,”  “You  can’t 
use  more  than  five  of  my  people  on  the  team,”  and  “I  can’t  afford  to  have  it 
last  more  than  a  month.”  Constraints  identified  by  the  appraisal  input  must  be 
negotiated  between  the  sponsor  and  the  appraisal  team  leader. 
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1 .1 .2  Determine  Appraisal  Constraints  (continued) 


Optional 

Practices 

Implementation 

Guidance 


Page  11-16 


Document  the  rationale  for  choices  made  and  the  associated  tradeoffs  as  a 
resource  for  later  planning  and  future  appraisals. 


Practical  limitations  relating  to  time,  cost,  and  effort  are  clarified  and 
negotiated  in  the  context  of  other  requirements  the  sponsor  has.  The  business 
context  in  which  the  appraisal  is  conducted  drives  choices  that  the  appraisal 
team  leader  needs  to  make.  Appraisals  should  not  be  conducted  in  isolation 
from  other  activities  relating  to  process  management  and  improvement.  The 
needs  of  related  stakeholders,  be  they  acquisition  organizations  or  division 
heads  managing  the  Engineering  Process  Group,  often  place  requirements  on 
the  conduct  of  the  appraisal. 
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1 .1 .2  Determine  Appraisal  Constraints 


GSS  and  CPM  Implementation  Guidance 

When  using  the  MDD  for  Government  source  selection  and  contract  process  monitoring,  the 
following  should  be  considered: 

Guidance  for  The  purpose  of  this  activity  is  to  identify  the  appraisal  constraints.  Detailed 

GSS  and  CPM  planning  concerning  these  constraints  is  then  done  in  process  1.2,  Develop 

Appraisal  Plan.  The  degree  to  which  decisions  are  made  in  process  1.1, 
Analyze  Requirements,  versus  process  1.2  is  expected  to  vary  with  the  needs 
of  each  individual  procurement.  The  following  items  should  be  considered,  at 
a  minimum,  as  possible  sources  of  appraisal  constraints.  Subsequent  detailed 
planning  must  then  be  accomplished  for  all  of  these  items  in  process  1.2. 

Cost  Constraints 

The  total  cost  of  the  appraisal  includes  both  costs  of  the  sponsoring 
acquisition  organization  and  of  the  contractor  organization(s)  undergoing  the 
appraisals.  Costs  of  the  sponsoring  acquisition  organization’s  appraisal  team 
include  costs  for  personnel  effort,  travel,  and  training.  Note  that  the  personnel 
effort  considered  in  the  cost  must  include  preparation  for  the  appraisal  as  well 
as  performing  the  appraisal  and  reporting  the  results  (i.e.,  must  include  the 
effort  to  perform  all  required  SCAMPI  processes).  Costs  applicable  to  both 
the  appraisal  team  members  and  members  of  the  sponsoring  acquisition 
organization  who  are  not  part  of  the  appraisal  team  must  be  considered.  Other 
planning  decisions,  such  as  the  organizational  units  to  be  appraised,  the 
appraisal  scope,  and  the  locations  of  the  appraisals,  will  affect  the  appraisal 
costs. 

When  an  appraisal  is  performed  during  the  source  selection  period,  the  costs 
to  the  organization(s)  undergoing  the  appraisals  are  assumed  by  the  appraised 
organization(s).  However,  there  are  circumstances  under  which  appraisal  costs 
will  be  charged  to  the  contract.  Program  cost  estimates  must  include  sufficient 
funds  to  cover  the  costs  of  any  appraisals  charged  to  the  contract.  Examples  of 
these  circumstances  are  as  follows: 

•  All  costs  for  CPM  appraisals  after  contract  award  are  direct  contract  costs. 

•  In  a  multi-phase  procurement  where  a  sequence  of  down-selections  takes 
place,  a  SCAMPI  appraisal  may  be  performed  during  a  preceding  contract 
period  with  the  appraisal  report  provided  as  input  to  the  next  source 
selection.  (Performing  the  appraisal  outside  of  the  source  selection  is  an 
approach  sometimes  taken  to  alleviate  the  source  selection  schedule 
constraints.) 


Continued  on  next  page 


CMU/SEI-2002-HB-002 


Page  11-17 


1 .1 .2  Determine  Appraisal  Constraints 


GSS  and  CPM  Implementation  Guidance  (continued) 

Guidance  for  Schedule  Constraints 

GSS  and  CPM,  For  GSS  SCAMPI  appraisals,  the  appraisal  must  fit  into  the  source  selection 

cont.  schedule.  For  streamlined  acquisition  schedules,  this  can  be  an  especially 

severe  constraint.  (An  example  source  selection  timeline  is  shown  in 
Appendix  E,  Figure  HI-1.)  For  GSS  appraisals  performed  outside  the  time 
frame  of  the  source  selection  but  still  a  factor  in  the  overall  source  selection 
award,  the  appraisals  must  be  scheduled  to  be  consistent  with  overall  program 
and  source  selection  schedules.  In  this  situation,  the  final  reports  for  all 
organizations  appraised  must  be  completed  in  time  to  be  provided  as  input  to 
the  source  selection.  For  CPM  SCAMPI  appraisals,  one  needs  to  consider  the 
frequency  and  scheduling  of  appraisals  throughout  the  contract  period  of 
performance  to  provide  the  minimum  impact  on  the  program  while  meeting 
the  contract  and  appraisal  objectives. 

Awards  With  and  Without  Discussions  (GSS  Usage  Only) 

A  decision  needs  to  be  made  as  to  whether  performing  a  SCAMPI  appraisal 
will  constitute  discussions  for  the  source  selection.  This  decision  must  be 
made  by  the  Contracting  Officer  or  a  legal  or  procurement  official.  Many 
streamlined  acquisitions  have  award  without  discussions  as  a  goal. 

If  the  appraisal  is  to  be  performed  in  a  source  selection  that  is  planning  to 
award  without  discussions,  or  if  the  appraisal  is  to  be  performed  before 
discussions  are  opened,  there  may  be  restrictions  on  the  type  of 
communication  that  can  occur  between  the  appraisal  team  and  the  bidding 
organization  undergoing  the  appraisal.  Examples  of  such  restrictions  are 
prohibitions  of  discussions  of  the  proposal  or  program  under  bid  and 
restriction  of  communication  to  the  past  projects  under  appraisal.  There  may 
also  be  restrictions  as  to  what  can  be  communicated  to  the  bidding 
organization  undergoing  the  appraisal  during  the  process  of  validating 
preliminary  findings.  One  way  to  avoid  these  problems  is  to  perform  the 
appraisal  outside  of  the  source  selection  using  an  appraisal  team  that  is  not 
part  of  the  source  selection  team,  with  the  final  reports  input  into  the  source 
selection.  If  discussions  are  planned  to  be  held  during  the  source  selection, 
another  option  (depending  upon  the  source  selection  schedule  and  the  number 
of  appraisals  to  be  performed)  is  to  perform  the  appraisal  after  discussions  are 
opened. 

See  the  Implementation  Guidance:  Focus  Groups  and  Guidance  for  GSS 
sections  in  activity  2.2.3,  Validate  Practice  Implementation  Gaps,  for  optional 
techniques  for  the  validation  of  preliminary  findings  during  government 
source  selections. 
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GSS  and  CPM  Implementation  Guidance  (continued) 

Placement  in  the  Source  Selection  Evaluation  Structure  (GSS  Usage  Only) 

For  source  selection,  decisions  must  be  made  as  to  how  the  appraisal  results 
will  be  included  in  the  source  selection  evaluation  structure.  Three  common 
placements  for  appraisal  results  are  (1)  as  part  of  the  Technical  Evaluation,  (2) 
as  part  of  the  Management  Evaluation,  and  (3)  as  part  of  the  Past  Performance 
Evaluation.  Different  acquisition  organizations  may  have  different  factor  and 
subfactor  structures  and  different  philosophies  as  to  which  placement  is 
appropriate  for  the  SCAMPI  appraisal.  Whichever  placement  is  chosen, 
however,  the  SCAMPI  appraisal  should  be  placed  so  that  it  is  a  significant 
consideration  in  the  source  selection  evaluation,  commensurate  with  the 
degree  of  risk  of  immature  systems  and  software  engineering  processes  on  the 
program  and  with  being  able  to  achieve  the  appraisal  objectives. 

Once  the  overall  placement  of  the  appraisal  in  the  source  selection  evaluation 
structure  is  determined,  the  specific  details  for  how  the  SCAMPI  appraisal 
results  are  to  be  related  to  the  source  selection  evaluation  criteria  must  be 
defined  (e.g.,  how  a  SCAMPI  strength  or  weakness  relates  to  a  source- 
selection  technical  or  management  strength  or  weakness,  how  the  SCAMPI 
results  relate  to  the  evaluation  of  proposal  risk,  how  the  SCAMPI  results 
relate  to  the  past  performance  ratings,  how  SCAMPI  weaknesses  or  risks  will 
be  handled  by  Evaluation  Notices  [ENs]).  Wherever  the  SCAMPI  is  placed  in 
the  GSS  structure,  it  is  imperative  that  coverage  of  the  SCAMPI  be  included 
in  the  GSS  evaluation  criteria  for  that  area. 

Reporting  Constraints 

Reporting  constraints  can  apply  to  both  GSS  and  CPM  usage.  For  GSS,  what 
can  be  presented  to  the  bidding  organization  undergoing  appraisal  for 
preliminary  and  final  findings  and  when  those  findings  can  be  presented  might 
be  restricted. 

Disclosure  of  final  findings  to  the  offerors  is  usually  delayed  until  after 
contract  award  has  been  made.  Agreement  with  the  Contracting  Officer  and 
other  source  selection  authorities  must  be  obtained  in  order  to  be  able  to 
present  the  details  of  the  final  findings  to  the  appraised  organization  after 
contract  award.  It  may  only  be  possible  to  disclose  the  source  selection  ratings 
(e.g.,  color  ratings,  strengths  and  weaknesses)  at  a  higher  level  in  the  source 
selection  technical,  management,  or  past  performance  structure  (rather  than 
being  able  to  present  the  findings  for  each  goal  of  each  appraised  process 
area).  Alternatively,  it  may  only  be  possible  to  disclose  the  detailed  SCAMPI 
findings  (e.g.,  strengths  and  weaknesses  for  each  appraised  process  area)  but 
not  the  overall  source  selection  scoring  (e.g.,  color  ratings,  strengths  and 
weaknesses).  For  an  ARC  Class  A  appraisal,  however,  at  a  minimum  the 
summary  strengths  and  weaknesses  for  each  appraised  process  area  must  be 
presented  to  both  the  appraised  organization  and  the  appraisal  sponsor. 


Guidance  for 
GSS  and  CPM, 
cont. 
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GSS  and  CPM  Implementation  Guidance  (continued) 

Guidance  for  Appraisal  ratings  (e.g.,  goal  and/or  maturity  level  or  capability  level  ratings) 

GSS  and  CPM,  planned  for  and  generated  by  the  team  must  be  given  to  the  appraisal  sponsor, 

cont.  but  are  not  required  to  be  given  to  the  appraised  organization. 

Similarly,  the  process  of  validating  preliminary  findings  while  on  site  may  be 
restricted.  It  may  be  possible  to  use  on-site  interviews  in  a  validation  mode  for 
preliminary  observations/findings.  Another  validation  option  is  to  use  ENs  to 
send  appraisal  weaknesses  and  risks  (derived  from  preliminary 
observations/findings)  to  the  bidding  organization  undergoing  appraisal  for 
response.  This  latter  option  can  only  be  used  if  discussions  are  opened.  Note 
that  at  least  one  validation  session  (i.e.,  one  form  of  validation)  is  required  for 
an  ARC  Class  A  appraisal. 

It  is  important  to  work  with  the  Contracting  Officer  to  define  in  detail  the 
constraints  on  presenting  SCAMPI  findings  (both  preliminary  and  final)  to  the 
appraised  organizations.  For  CPM,  presentation  of  final  findings  to  the 
appraised  organization  is  not  generally  a  problem.  The  timing  of  this 
presentation,  however,  may  need  to  be  negotiated.  If  the  appraisal  will  be  used 
to  determine  incentive  or  award  fees,  however,  additional  restrictions  may  be 
placed  on  the  process.  (See  activities  2.2.3  and  3.1.1  for  additional 
information  on  alternatives  for  preliminary  and  final  findings,  respectively.) 

In  addition,  it  is  important  to  determine  whether  any  constraints  exist  for  other 
SCAMPI  reporting  requirements  (e.g.,  the  appraisal  record,  information 
reported  to  the  CMMI  Steward).  Reporting  constraints  for  CPM  usage  are  not 
likely  to  be  as  severe  as  those  for  GSS  usage,  but  some  may  still  exist.  Source- 
selection-sensitive  information  may  not  be  releasable  at  any  time  or  may  only 
be  releasable  after  contract  award  (e.g.,  offerors’  names,  number  of  offerors). 
Other  information  may  be  classified  (e.g.,  program  names).  Reports  to  the 
CMMI  Steward  could  be  delayed  until  after  contract  award  and  sanitized  to 
conform  to  applicable  acquisition  regulations/policies.  Close  coordination 
with  the  program  manager  and  Contracting  Officer  is  necessary  to  fully  define 
all  reporting  constraints  for  the  appraisal.  Note  that  if  the  reporting  constraints 
are  too  restrictive  to  satisfy  SCAMPI  reporting  requirements,  an  ARC  Class  B 
or  Class  C  appraisal  may  need  to  be  used. 
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GSS  and  CPM  implementation  Guidance  (continued) 

Guidance  for  Appraisal  Team  Constraints 

GSS  and  CPM,  For  GSS  SCAMPI  usage,  the  appraisal  team  may  or  may  not  be  part  of  the 
cont.  Source  Selection  Evaluation  Board  (SSEB).  There  are  advantages  and 

disadvantages  to  each  option.  While  an  appraisal  team  that  is  part  of  the  SSEB 
may  have  more  influence  on  the  source  selection  results,  it  may  be  difficult  to 
find  enough  qualified  appraisal  team  members  that  meet  the  criteria  for  being 
on  the  SSEB  (e.g.,  the  SSEB  may  be  composed  of  only  Government 
personnel).  For  some  acquisition  organizations,  an  appraisal  team  can  consist 
of  advisors  for  the  source  selection  while  not  being  part  of  the  SSEB  (e.g.. 
Federally  Funded  Research  and  Development  Center  (FFRDC)  personnel  can 
be  advisors  for  the  source  selection).  In  this  case,  the  appraisal  team  can 
provide  direct  input  into  the  source  selection  and  can  still  be  quite  influential. 
If  the  appraisal  is  performed  outside  of  the  source  selection,  fewer  restrictions 
will  apply  to  the  members  of  the  appraisal  team,  but  the  team  may  not  be  able 
to  interface  directly  with  the  source  selection  participants.  Whether  the 
appraisal  is  performed  inside  or  outside  of  source  selection,  the  appraisal  team 
will  need  to  satisfy  the  conflict  of  interest  requirements  of  the  acquisition 
organization.  In  the  case  where  the  SCAMPI  team  members  are  full  SSEB 
members  and  the  appraisal  is  performed  during  the  source  selection,  the 
source  selection  schedule  may  need  to  be  lengthened  to  accommodate  their 
multiple  roles. 

Another  consideration  is  whether  or  not  multiple  appraisal  teams  will  be 
fielded  to  perform  the  appraisals  concurrently.  Fielding  multiple  appraisal 
teams  is  one  mechanism  for  dealing  with  the  schedule  constraints  of  source 
selection,  but  may  also  be  used  for  CPM  if  a  number  of  different  organizations 
must  be  appraised.  Consideration  must  be  given  here  to  the  risk  of  differences 
in  how  the  appraisals  are  conducted  and  also  to  whether  enough  personnel  that 
meet  all  of  the  appraisal  team  member  requirements  can  be  found  to  staff 
multiple  teams. 

•-  For  both  GSS  and  CPM  usage,  the  Contracting  Officer  may  have  additional 
constraints  on  appraisal  team  membership.  In  addition,  the  Contracting 
Officer  may  have  other  restrictions  placed  on  conducting  the  appraisal,  such 
as  the  necessity  for  a  Contracting  Officer  to  be  present  at  all  site  visits. 

Location  Constraints 

Constraints  on  the  location  of  the  site  visits  will  be  affected  by  the 
organizational  scope  of  the  appraisal  and  will  affect  the  appraisal  cost  and 
schedule.  There  are  a  number  of  options  for  site  visits.  For  example,  they  may 
be  held 

•  at  each  appraised  organization’s  location 

•  at  one  location  per  offeror,  selected  by  the  sponsoring  acquisition 
organization  or  by  the  offeror 

•  at  multiple  locations  for  a  geographically  dispersed  bidding  team 
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GSS  and  CPM  Implementation  Guidance  (continued) 

Appraisal  Scope  Constraints 

Any  constraints  on  the  scope  of  the  SCAMPI  appraisal  must  be  identified. 
This  includes  constraints  on  the  reference  model  scope  and  on  the 
organizational  unit(s)  to  be  appraised.  More  information  on  appraisal  scope 
considerations  is  given  in  the  guidance  for  activity  1.1.3,  Determine  Appraisal 
Scope.  Note  that  in  that  activity,  the  full  appraisal  scope  is  determined.  Here 
only  constraints  on  the  appraisal  scope  are  identified. 

SCAMPI  Method  Constraints 

Any  constraints  that  may  preclude  an  ARC  Class  A  appraisal  from  being 
performed  must  be  identified.  This  includes  required  SCAMPI  processes  and 
activities  that  are  prohibited  from  being  performed  by  the  rules  of  the 
procurement.  It  also  includes  SCAMPI  processes  and  activities  (e.g.,  goal 
ratings)  that  must  be  tailored  in  such  a  way  that  requirements  of  the  SCAMPI 
method  are  not  met  in  order  to  meet  the  rules  of  the  procurement.  In  such  a 
case,  it  may  be  possible  to  define  an  ARC  Class  B  or  Class  C  appraisal  that 
fits  within  the  constraints. 

Consistency  of  Constraints 

When  exiting  from  this  activity,  it  is  essential  that  the  set  of  constraints  is  self- 
consistent  (i.e.,  not  internally  contradictory).  Multiple  iterations  through  this 
activity  may  be  required  to  obtain  a  self-consistent  set  of  constraints. 
Contradictory  constraints  need  to  be  resolved  for  a  successful  appraisal. 
Resolution  of  contradictory  constraints  may  preclude  performance  of  an  ARC 
Class  A  appraisal. 

Negotiation  of  Constraints 

This  activity  includes  the  negotiation  of  the  constraints  with  the  stakeholders. 
It  is  not  sufficient  to  obtain  concurrence  on  the  constraints  solely  from  the 
appraisal  sponsor.  Concurrence  must  also  be  obtained  from  the  program 
manager.  Contracting  Officer,  and  personnel  responsible  for  the  acquisition 
strategy  and  source  selection. 


Guidance  for 
GSS  and  CPM, 
cont. 
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Activity 

Description 


Required 

Practices 


The  appraisal  scope  consists  of  the  reference  model  scope  and  the 
organizational  scope  to  be  examined  during  the  appraisal.  The  model  scope 
must  be  determined  and  documented  early  in  the  planning  process,  using  the 
staged  representation  or  the  continuous  representation.  The  appraisal  team 
leader  is  responsible  for  ensuring  that  the  sponsor  makes  an  informed  choice 
regarding  the  PAs  included  in  the  scope  of  the  appraisal  and  the  model 
representation.  The  selection  of  appraisal  outputs  should  be  driven  by  the 
understanding  of  their  intended  use,  established  during  the  requirements 
analysis  activity,  and  may  dictate  some  selections  in  model  scope.  The 
organizational  scope  defines  the  bounds  of  the  organization  to  be  investigated 
during  the  appraisal.  Instantiations  (i.e.,  for  practices  implemented  by  projects, 
each  project;  for  practices  implemented  organization-wide,  the  instance)  are 
selected  as  representative  of  the  organization  and  the  contexts  in  which 
processes  are  implemented. 

Reconciling  the  interactions  between  model  scope  and  organization  scope  is 
an  important  part  of  this  activity.  A  particular  organization  scope  begets  a 
particular  model  scope;  a  particular  model  scope  requires  a  particular 
organization  scope. 


•  Determine  and  document  the  reference  model  scope  and  representation  to 
be  used  for  the  appraisal. 

•  Determine  and  document  the  organizational  unit  to  be  investigated  during 
the  appraisal. 


Continued  on  next  page 
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Parameters 
and  Limits 


Optional 

Practices 


The  reference  model  scope  includes  the  PAs  and  associated  maximum 
capability  level  and/or  maturity  level  that  will  be  investigated  by  the  appraisal 
team  (i.e.,  the  generic  goals  that  will  be  rated  for  each  PA  within  the  scope  of 
the  appraisal).  Note  that  the  selection  of  the  reference  model  representation 
should  have  been  discussed  during  the  setting  of  appraisal  objectives,  because 
the  representation  selected  may  impact  the  achievability  of  these  objectives. 

The  model  scope  of  the  appraisal  must  encompass  at  least  one  PA.  All  generic 
goals  and  specific  goals  up  to  and  including  the  target  capability  level  arid/or 
maturity  level  for  each  selected  PA  must  be  rated;  individual  goals  within  a 
PA  cannot  be  excluded. 

Instantiations  must  be  selected  that  are  representative  of  the  implemented 
processes  and  functional  areas  being  investigated  within  the  organizational 
unit,  and  that  operate  within  a  coherent  process  context  (see  glossary  for 
definition).  This  is  also  sometimes  known  as  the  organizational  scope  of  the 
appraisal.  The  rationale  for  selecting  these  elements  as  representative  of  the 
organizational  unit  should  be  documented. 

Typically,  the  organizational  unit  will  be  specified  in  such  a  manner  that  (a)  at 
least  two  instances  of  the  processes  being  investigated  are  available  as  sources 
of  objective  evidence  and  (b)  a  representative  coverage  of  the  life  cycles  in 
use  within  the  organization  is  obtained.  Selection  of  instantiations  within  the 
organizational  unit  may  be  accomplished  through  a  survey  form,  or  through 
summarizing  information  learned  from  discussions  with  members  of  the 
organization.  For  processes  enacted  at  the  organization  level  (such  as 
Organizational  Training),  multiple  instances  are  not  required. 

The  representative  instantiations  to  be  investigated  during  the  appraisal  will 
also  drive  the  selection  of  participants  needed  to  provide  sources  of  objective 
evidence.  An  initial  determination  of  appraisal  participants,  by  name  and  role, 
should  be  negotiated  with  the  appraisal  sponsor  and/or  the  senior  site  manager 
as  part  of  the  early  determination  of  organizational  scope.  This  will  be  refined 
later  during  detailed  appraisal  planning. 


Use  broad-based  survey  instruments  or  a  Practice  Implementation  Indicator 
(PII)  database  to  characterize  the  population  of  projects  or  divisions  in  an 
organization  before  determining  the  organizational  scope  of  the  appraisal. 
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Implementation 

Guidance 


There  are  two  primary  parameters  of  the  appraisal  enactment  that  contribute 
significantly  to  the  resulting  cost  (in  terms  of  effort):  the  PA  scope  of  the 
CMMI  model  encompassed  by  the  appraisal  and  the  number  and  size  of 
projects  selected.  While  other  parameters  contribute  to  the  cost  and  schedule 
of  an  appraisal,  these  two  scope  parameters  provide  the  greatest  opportunity  to 
shape  the  magnitude  of  the  appraisal.  SCAMPI  requires  that  findings  for  the 
organizational  unit  be  derived  from  objective  evidence  on  the  implementation 
of  practices  collected  from  each  of  the  organizational  process  instantiations 
included  in  the  appraisal.  The  size  and  number  of  instantiations  investigated 
should  be  selected  to  form  a  valid  sample  of  the  organizational  unit  to  which 
the  results  will  be  attributed. 

Clearly,  a  broadly  defined  organizational  unit  (e.g.,  a  multi-national 
enterprise)  will  require  collecting  and  analyzing  significantly  more  objective 
evidence  than  a  narrowly  defined  organizational  unit  (e.g.,  a  specific  product 
line  within  a  specific  business  unit  at  a  single  geographical  location). 

The  organization  to  which  appraisal  results  will  be  attributed  should  be 
described  accurately  in  all  statements  made  by  the  appraisal  team  leader  and 
sponsor.  It  is  the  responsibility  of  the  appraisal  team  leader  to  understand  the 
larger  organizational  context  in  which  the  appraised  organizational  unit 
resides.  Familiarity  with  the  nature  of  departmental  structures,  matrixed 
subject  matter  expert  groups,  integrated  product  teams,  program  and  project 
groupings,  or  product  line  implications  that  may  affect  the  interpretation  of 
appraisal  outcomes  will  aid  in  obtaining  this  understanding. 

The  appraisal  team  leader  should  work  with  representatives  from  the 
organization  to  document  a  clear  statement  of  the  model  and  organizational 
scope  of  the  appraisal.  The  model  scope  should  be  documented  using  a  list  of 
PAs  to  be  included  in  the  appraisal,  as  well  as  the  model  components  to  be 
rated  by  the  appraisal  team.  The  organizational  scope  of  the  appraisal  should 
be  documented  in  the  clearest  terms  possible,  given  the  nature  of  the 
organizational  structure  in  place.  It  is  often  difficult  to  specify  unambiguous 
boundaries  without  resorting  to  naming  individual  people  in  some 
organizations.  Information  about  the  organizational  unit  should  be 
documented  in  a  way  that  allows  future  appraisal  sponsors  to  replicate 
(exactly)  the  scope  of  the  organization  appraised.  This  information  should  be 
in  the  appraisal  plan,  and  used  (in  summary  form  if  needed)  in  briefing  the 
appraisal  team  and  appraisal  participants. 
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GSS  and  CPM  Implementation  Guidance 

When  using  the  MDD  for  Government  source  selection  and  contract  process  monitoring,  the 

following  should  be  considered: 

Guidance  for  Reference  Model  Scope 

GSS  and  CPM  For  a  GSS  or  CPM  SCAMPI  appraisal,  a  principal  consideration  is  whether  a 
maturity  level  rating  or  capability  ratings  are  required.  The  need  for  these 
types  of  ratings  in  an  acquisition  are  frequently  driven  by  specific  policies  or 
regulations  that  apply  to  the  acquisition  organization.  The  need  for  maturity 
level  or  capability  level  ratings  dictate  which  model  representation  is  used 
(staged  or  continuous,  respectively).  In  addition,  the  need  for  a  maturity  level 
rating  or  equivalent  staging  for  continuous  representation  dictates  what  PAs 
must  be  appraised  (see  “Model  Tailoring  Criteria  for  Benchmarking”  in  the 
“Using  CMMI  Models”  chapter  of  any  CMMI  Version  1.1  model  for 
minimum  scope  requirements).  More  flexibility  in  the  definition  of  the  model 
scope  is  possible  if  maturity  level  ratings  or  equivalent  staging  for  continuous 
representation  are  not  required. 

It  should  be  noted  that  in  the  GSS  environment  the  model  scope  for  the 
appraisals  should  be  identical  for  all  offerors.  This  would  enable  the  SCAMPI 
appraisal  criteria  to  be  consistent  with  enabling  the  acquisition  organization  to 
evaluate  individual  offerors  using  a  common  set  of  PAs  and  their  resulting 
strengths  and  weaknesses. 

When  selecting  the  disciplines  and  PAs  to  be  covered  by  the  appraisal, 
consideration  should  be  given  to  program  requirements  and  risks.  To  reduce 
reference  model  scope,  focus  should  be  on  those  disciplines  and  PAs  in  which 
immature  processes  are  considered  to  be  of  highest  risk  to  the  program. 

For  CPM  SCAMPI  usage,  frequently  a  series  of  appraisals  is  performed 
throughout  the  contract  period.  In  this  case,  the  scope  of  each  appraisal  must 
be  determined.  One  way  to  accomplish  a  series  of  CPM  appraisals  is  to  have 
each  cover  the  same  reference  model  scope,  with  enough  time  between 
appraisals  to  allow  for  observable  process  improvements.  Another  way  to 
accomplish  a  series  of  CPM  appraisals  is  to  have  a  series  of  more  frequent 
appraisals  in  which  each  appraises  a  different  subset  of  PAs  on  a  rotating 
basis.  A  third  method  is  to  use  findings  from  previous  appraisals  as  input  in 
determining  model  scope  for  subsequent  appraisals  (e.g.,  previously 
unsatisfied  goals,  previously  identified  weaknesses,  areas  addressed  in  a 
process  improvement  plan,  or  areas  related  to  incentive/award  fee 
determination  based  on  process  improvement). 
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GSS  and  CPM  Implementation  Guidance  (continued) 

Guidance  for  Organizational  Scope 

GSS  and  CPM,  The  definition  of  the  organizational  scope  of  the  appraisal  depends  on  the 
cont.  appraisal  objectives,  the  nature  of  the  program,  and  the  number  and  structure 

of  the  bidding  teams  (for  GSS)  or  contractor  teams  (for  CPM).  The  SCAMPI 
method  is  designed  to  handle  a  wide  range  of  possibilities  for  the 
organizational  unit. 

Note:  The  SCAMPI  method  does  not  specify  a  precise  number  for  the  number 
of  projects,  but  allows  the  appraisal  sponsor  and  Lead  Appraiser  to  define  the 
organizational  unit.  When  moving  from  project-specific  processes  to 
organizational  processes  that  cross  project  and  supporting  organizations’ 
boundaries,  this  definition  becomes  especially  critical.  The  paragraphs  that 
follow  describe  the  significant  GSS  and  CPM  issues  related  to  defining  the 
organizational  unit  to  be  appraised. 

For  GSS  usage,  the  structure  of  the  bidding  teams  may  not  be  known  until  the 
initial  objective  evidence  is  received.  Therefore,  the  precise  definition  of  the 
organizational  units  to  be  assessed  may  not  be  determinable  until  later  in  the 
planning  process.  This  activity,  however,  should  define  the  approach  to  be 
*  used  for  the  organizational  units  to  be  appraised. 

No  matter  how  the  organizational  unit  is  defined,  several  key  ground  rules 
should  be  observed: 

•  All  bidding  team  members  performing  significant  systems/software 
engineering  should  be  appraised.  Note  that  bidding  team  members  that 
perform  significant  amounts  of  systems/software  engineering  may  be 
subcontractors,  subcontractors  of  subcontractors,  or  intra-corporate 
organizations  of  the  prime  contractor  or  subcontractor  (i.e.,  bidding  team 
members  that  are  other  organizations,  such  as  other  business  units,  within 
the  prime  contractor’s  or  a  subcontractor’s  company). 

•  Questions  about  unique  or  different  processes  should  be  answered 
individually  by  the  bidding  team  members  using  those  processes. 

•  Even  when  a  bidding  team  proposes  common  processes,  evidence  should 
be  provided  by  the  individual  bidding  team  members. 

Possible  choices  for  organizational  scope  include  the  following: 

•  Appraise  prime  contractors  only. 

This  is  appropriate  for  small  programs  where  virtually  all  of  the 
systems/software  engineering  is  performed  by  the  prime  contractor,  with 
the  prime  contractor  possibly  contracting  with  vendors  for  commercial 
products.  For  a  larger  program  in  which  one  or  more  subcontractors 
perform  significant  amounts  of  systems/software  engineering,  this 
organizational  scope  would  not  be  appropriate. 
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GSS  and  CPM  Implementation  Guidance  (continued) 

•  Appraise  prime  contractors  and  all  significant  systems 
engineering/software  engineering  teams  as  individual  organizations. 

For  large  programs,  especially  for  source  selections  with  multiple 
offerors,  this  organizational  scope  could  result  in  many  appraisals  to  be 
performed.  This  organizational  scope  is  appropriate  for  programs  in  which 
the  prime  contractor  and  its  subcontractors  function  nearly  autonomously. 
However,  this  type  of  organizational  scope  does  not  cover  the  necessary 
interactions  across  team  boundaries  that  are  usually  high  process  risk 
areas. 

•  Appraise  the  bidding  team  (for  GSS)  or  contractor  team  (for  CPM)  as  a 
single  organizational  unit. 

This  approach  focuses  on  how  the  team,  as  a  cohesive  unit,  plans  to  do 
business  (for  GSS)  or  does  business  (for  CPM),  rather  than  focusing  on 
the  individual  capabilities  of  the  team  members.  This  type  of 
organizational  unit  is  appropriate  for  programs  in  which  the 
systems/software  engineering  performed  by  different  team  members  must 
be  integrated  to  accomplish  program  objectives.  For  CPM  usage,  evidence 
will  exist  as  to  how  well  the  combined  team  capabilities  work.  For  GSS 
usage  in  a  multi-phase  contracting  approach,  evidence  may  exist  from  a 
preceding  contract  period  as  to  how  well  the  combined  team  capabilities 
work.  However,  in  many  GSS  usages,  no  historical  evidence  of  the  team 
capabilities  will  be  available  and  evidence  must  be  collected  from  the 
team  members. 

For  GSS  usage,  full  determination  of  the  organizational  scope  will  usually 
require  information  to  be  provided  by  the  bidding  teams  prior  to  the  appraisal 
concerning  the  identification,  location(s),  and  assigned  responsibilities  of  the 
team  members  and  the  team  structure.  Any  needed  communication  with  the 
bidding  teams  is  usually  done  via  the  Contracting  Officer,  rather  than  direct 
communication  between  the  appraisal  POC  or  Lead  Appraiser  and  the  bidding 
teams.  Details  needed  to  finalize  the  selection  of  projects  to  be  examined  and 
people  to  be  interviewed  may  not  be  knowable  until  the  initial  objective 
evidence  is  obtained.  (See  activity  1.4.3,  Obtain  Initial  Objective  Evidence.) 

Treatment  of  the  Program  Under  Bid  (for  GSS  only) 

For  GSS  SCAMPI  usage,  part  of  this  activity  should  be  the  determination  of 
how  to  treat  the  program  under  bid  in  the  appraisal.  In  a  multi-phase 
procurement,  for  example,  there  may  be  a  current  or  prior  contract  with  the 
bidding  team  from  which  evidence  can  be  examined.  In  this  case,  it  is 
recommended  that  that  contract  be  a  required  project  for  the  appraisal.  For  a 
CPM  SCAMPI  usage,  only  one  project  needs  to  be  examined,  that  is,  the 
project  under  contract. 


Guidance  for 
GSS  and  CPM, 
cont. 
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GSS  and  CPM  Implementation  Guidance  (continued) 

Guidance  for  Consistency  Considerations 

GSS  and  CPM,  The  defined  appraisal  scope  must  not  only  be  internally  consistent,  it  must 

cont.  also  be  consistent  with  the  appraisal  objectives  defined  in  activity  1.1.1,  the 

appraisal  constraints  defined  in  activity  1.1.2,  and  the  appraisal  outputs 
defined  in  activity  1.1.4.  Of  particular  concern  here  is  the  ability  to  carry  out 
the  SCAMPI  appraisal  with  the  defined  scope  and  outputs  within  the 
documented  cost  and  schedule  constraints. 

Guidance  for  The  appraisal  team  should  also  be  briefed  on  the  program  in  source  selection 

GSS  and  be  thoroughly  familiar  with  any  peculiar  domain-driven  requirements  for 

the  appraisal.  Knowledge  of  the  acquisition  program  may  drive  areas  that 
require  additional  or  reduced  focus  during  the  appraisal.  For  example,  for 
spiral  development  of  unprecedented  performance  features  or  technologies, 
requirements  development  and  management  may  be  identified  for  special 
attention  during  the  appraisal. 
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1 .1 .4  Determine  Outputs 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Identify  the  specific  appraisal  outputs  to  be  produced.  Some  appraisal  outputs 
are  required  and  additional  outputs  are  tailorable  (see  Parameters  and  Limits 
and  Optional  Practices). 

Obtain  unambiguous  answers  to  the  following  questions: 

•  What  ratings  will  be  generated  during  the  appraisal? 

•  Will  a  final  report  be  written  to  document  appraisal  results? 

•  Will  recommendations  on  how  to  address  specific  findings  be  generated 

and  reported?  *  • 


•  Review  required  SCAMPI  outputs  with  the  appraisal  sponsor.  Review  and 
select  optional  SCAMPI  outputs  with  the  appraisal  sponsor. 

•  Determine  the  recipients  of  appraisal  outputs. 


Required  SCAMPI  outputs  include 

•  Appraisal  Record  (see  activity  3.2.2,  Generate  Appraisal  Record) 

•  Appraisal  Disclosure  Statement  (see  activity  2.4.4,  Document  Appraisal 
Results) 

•  CMMI  Steward  Data  (see  activity  3.2.3,  Provide  Appraisal  Feedback  to 
CMMI  Steward) 

As  stated  in  the  ARC,  at  least  all  the  goals  for  the  process  area  or  areas 
investigated  by  the  team  must  be  rated,  although  the  choice  may  be  made  to 
not  disclose  the  ratings  to  anyone  other  than  the  appraisal  sponsor. 

At  a  minimum,  the  sponsor  gets  the  following  appraisal  outputs: 

•  final  findings,  including  statements  of  strengths  and  weaknesses 
documented  by  the  team  for  every  PA  investigated 

•  all  ratings  planned  for  and  generated  by  the  team 

Decisions  reached  on  appraisal  outputs,  including  what  ratings  will  be 
reported,  are  documented  in  the  appraisal  input. 


Continued  on  next  page 
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1 .1 .4  Determine  Outputs  (continued) 


Optional  The  appraisal  sponsor  may  request  that  additional  rating  outputs  be  generated 

Practices  as  a  result  of  the  appraisal.  Typical  rating  outputs  that  might  be  selected 

include 

•  maturity  level  and/or  capability  level  ratings 

•  PA  Satisfaction/Capability  Level  Profiles 

•  practice  ratings 

•  an  option  to  use  “partially  satisfied”  as  a  rating  assigned  to  a  PA 

•  15504  Process  Profile 

•  discipline-specific  ratings  (e.g.,  SE  or  SW) 

•  project-level  findings  or  ratings 

•  other  (non-typical)  outputs  desired 

Many  of  these  optional  appraisal  outputs  are  discussed  further  in  process  2.4, 
Generate  Appraisal  Results. 

The  sponsor  may  also  request  that  other  products  be  generated  as  appraisal 
outputs.  Typical  products  that  might  be  requested  include  (see  activity  3.1.3, 
Plan  for  Next  Steps): 

•  Appraisal  Final  Report 

•  Recommendations  for  taking  action  upon  the  appraisal  results 

•  Process  improvement  action  plan 


Implementation  Goal  satisfaction  ratings  for  both  specific  goals  and  generic  goals  of  the  PAs 
Guidance  within  the  scope  of  the  appraisal  are  a  minimum  requirement.  Capability 

and/or  maturity  level  ratings  are  optional.  There  is  no  requirement  to  report 
the  ratings  to  the  appraisal  participants  even  though  ratings  are  performed. 
The  sponsor  has  sole  authority  to  decide  (in  advance)  what  ratings  will  or  will 
not  be  reported,  and  to  whom  they  will  be  reported. 

While  statements  of  findings  are  a  required  output  of  the  method,  creating  a 
written  report  that  elaborates  on  the  findings  is  optional.  The  sponsor  should 
decide  if  resources  are  to  be  spent  creating  this  artifact.  Similarly,  the  task  of 
creating  recommendations  to  address  issues  uncovered  in  the  appraisal  may 
require  expertise  that  is  not  represented  on  the  appraisal  team  in  some  cases. 
The  characteristics  of  the  appraised  organization  and  the  constraints  that  shape 
its  improvement  program  should  be  carefully  considered  when  making 
process  improvement  recommendations. 
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1 .1 .4  Determine  Outputs 


GSS  and  CPM  Implementation  Guidance 

When  using  the  MDD  for  Government  source  selection  and  contract  process  monitoring,  the 

following  should  be  considered: 

Guidance  for  Agreements  on  appraisal  outputs  must  be  negotiated  with  the  appraisal 

GSS  and  CPM  sponsor  and  documented  in  the  appraisal  input.  However,  for  GSS  and  CPM 

usage,  it  is  not  sufficient  to  obtain  concurrence  solely  from  the  appraisal 
sponsor.  For  these  uses,  it  is  especially  important  to  also  obtain  concurrence 
on  appraisal  outputs  from  the  program  manager,  Contracting  Officer,  and 
personnel  responsible  for  the  acquisition  strategy  and  source  selection.  This  is 
especially  true  for  outputs  that  are  to  be  provided  to  anyone  other  than  the 
appraisal  sponsor. 

Of  particular  concern  for  GSS  or  CPM  is  the  CMMI  Steward  Data,  which 
Lead  Appraisers  are  required  to  submit.  Permission  to  disclose  this  data  to  the 
CMMI  Steward  must  be  obtained  from  the  Contracting  Officer.  Convincing 
the  Contracting  Officer  that  submission  of  this  data  will  not  compromise 
confidentiality  of  GSS  or  CPM  results  is  an  important  task  for  the  appraisal 
POC  and  Lead  Appraiser.  Reports  to  the  CMMI  Steward  could  be  delayed 
until  after  contract  award  and  sanitized  to  conform  to  applicable  acquisition 
regulations/policies.  Note  that  reporting  to  the  CMMI  Steward  is  required  for 
an  ARC  Class  A  appraisal. 

Documentation  of  final  findings  is  always  required  for  a  SCAMPI  appraisal. 
However,  final  reports  and  recommendation  reports  are  optional.  Final  reports 
documenting  the  appraisal  results  in  more  detail  than  the  final  findings  report 
should  be  prepared  for  both  source  selection  and  contract  process  monitoring 
to  fully  document  the  rationale  for  the  findings.  In  a  streamlined  GSS, 
however,  schedule  constraints  might  reduce  the  contents  of  such  a  final  report. 
For  GSS,  recommendations  on  how  to  address  specific  findings  will  usually 
not  be  allowed.  However,  such  recommendations  are  usually  acceptable,  and 
even  desirable,  for  CPM. 

For  GSS,  the  outputs  of  the  appraisal  need  to  support  the  source  selection 
evaluation  criteria.  For  CPM,  the  outputs  may  need  to  support,  for  example, 
development  and  monitoring  of  a  risk  mitigation  plan  or  meeting  planned 
measurement  goals  (target  capability  profile,  target  maturity  level,  number  of 
weaknesses  resolved).  The  format  and  content  of  the  final  reports  and 
recommendation  reports,  if  any,  must  be  based  on  the  needs  of  the  particular 
GSS  or  CPM  situation  (e.g.,  for  the  SCAMPI  results  to  be  appropriately 
considered  in  the  source  selection  evaluation  or  in  award  fee  evaluation  for 
CPM). 
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1 .1 .5  Obtain  Commitment  to  Appraisal  Input 


Activity 

Description 

Required 

Practices 


Parameters 
and  Limits 


The  appraisal  sponsor  formally  approves  the  appraisal  input,  and  this  set  of 
information  is  placed  under  change  management. 


•  Record  required  information  in  the  appraisal  input  record. 

•  Obtain  sponsor  approval  of  the  appraisal  input  record. 

•  Manage  changes  to  the  appraisal  input,  obtaining  sponsor  approval  of 
changes. 


The  appraisal  input  may  be  generated  incrementally  throughout  planning,  but 
must  be  approved  prior  to  the  start  of  data  collection.  At  a  minimum,  the 
appraisal  input  shall  provide  the  information  needed  to  address  the  following: 

•  the  identity  of  the  appraisal  sponsor  and  the  relationship  of  the  sponsor  to 
the  organizational  unit  being  appraised 

•  the  appraisal  purpose,  including  alignment  with  business  objectives  (see 
activity  1.1.1) 

•  the  appraisal  reference  model  scope  (see  activity  1.1.3) 

•  the  organizational  unit  being  appraised  (see  activity  1.1.3) 

•  the  process  context,  which,  at  a  minimum,  includes 

organizational  unit  size  and  demographics 
application  domain,  size,  criticality,  and  complexity 

-  high-priority  characteristics  (e.g.,  time  to  market,  feature  richness, 
reliability)  of  the  products  and  services  of  the  organizational  unit 

•  appraisal  constraints  (see  activity  1.1.2),  which,  at  a  minimum,  include 

-  availability  of  key  resources  (e.g.,  staffing,  funding,  tools,  facilities) 
schedule  constraints 

the  maximum  amount  of  time  to  be  used  for  the  appraisal 
specific  PAs  or  organizational  entities  to  be  excluded  from  the 
appraisal 

the  maximum,  minimum,  or  specific  sample  size  or  coverage  desired 
for  the  appraisal 

ownership  of  appraisal  results  and  any  restrictions  on  their  use 
controls  on  information  resulting  from  a  confidentiality  agreement 
non-attribution  of  appraisal  data  to  associated  sources 

•  the  identity  of  the  CMMI  models  used  (version,  discipline,  and 
representation) 


Continued  on  next  page 
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1 .1 .5  Obtain  Commitment  to  Appraisal  Input  (continued) 


Parameters 
and  Limits 
(continued) 


Optional 

Practices 

Implementation 

Guidance 


•  the  identity  and  affiliation  of  the  Lead  Appraiser  who  is  to  be  the  appraisal 
team  leader  for  the  appraisal 

•  the  identity  and  affiliation  of  the  appraisal  team  members,  with  their 
specific  appraisal  responsibilities 

•  the  identity  (name  and  organizational  affiliation)  of  appraisal  participants 
and  support  staff,  and  their  specific  responsibilities  for  the  appraisal 

•  any  additional  information  to  be  collected  during  the  appraisal  to  support 
the  achievement  of  the  appraisal  objectives 

•  a  description  of  the  planned  appraisal  outputs  (see  activity  1.1.4), 
including  ratings  to  be  generated 

•  anticipated  follow-on  activities  (e.g.,  reports,  appraisal  action  plans,  re¬ 
appraisal) 

•  planned  tailoring  of  SCAMPI  and  associated  tradeoffs,  including  the 
sample  size  or  coverage  of  the  organizational  unit 

•  appraisal  usage  mode  (i.e..  Internal  Process  Improvement,  Supplier 
Selection,  Process  Monitoring) 


None. 


A  Lead  Appraiser’s  ability  to  build  and  maintain  commitment  from  the 
sponsor  and  the  members  of  the  sponsoring  organization  is  a  major  factor 
contributing  to  the  success  of  the  appraisal.  The  process  of  understanding  the 
requirements  and  constraints  should  yield  a  series  of  agreements  that  form  an 
input  to  the  appraisal  plan.  Based  on  the  judgment  of  the  appraisal  team 
leader,  these  agreements  may  be  covered  in  a  formal  (signed)  document  that 
forms  a  basis  for  future  activities.  More  typically,  the  appraisal  team  leader 
maintains  a  record  of  interactions  with  the  sponsor,  which  are  incorporated 
into  the  appraisal  plan  as  it  is  drafted. 

The  appraisal  team  leader  and  the  sponsor  should  have  verbal  agreement  on 
the  items  discussed  above,  and  these  items  should  be  documented  in  some 
way.  The  formality  of  the  documentation  may  range  from  simple  meeting 
minutes  maintained  by  the  appraisal  team  leader,  to  a  more  formal 
Memorandum  of  Understanding  or  other  vehicle  that  documents  agreements 
and  provides  traceability.  It  is  expected  that  the  appraisal  plan  will  be  used  to 
document  important  issues  pertaining  to  requirements. 
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1 .1 .5  Obtain  Commitment  to  Appraisal  Input 


GSS  and  CPM  Implementation  Guidance 

When  using  the  MDD  for  Government  source  selection  and  contract  process  monitoring,  the 
following  should  be  considered: 

Guidance  for  For  GSS  or  CPM  usage,  it  is  important  to  obtain  formal  agreement  to  the 

GSS  and  CPM  appraisal  input  from  not  only  the  appraisal  sponsor  but  also  the  program 

manager,  Contracting  Officer,  and  personnel  responsible  for  the  acquisition 
strategy  and  source  selection. 
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1.2  Develop  Appraisal  Plan 


Purpose 


Entry  Criteria 


Inputs 


Activities 


Outputs 


Outcome 


Exit  Criteria 


Document  the  results  of  appraisal  planning  including  the  requirements, 
agreements,  estimates,  risks,  method  tailoring,  and  practical  considerations 
(e.g.,  schedules,  logistics,  and  contextual  information  about  the  organization) 
associated  with  the  appraisal.  Obtain  and  record  the  sponsor’s  approval  of  the 
appraisal  plan. 


An  appraisal  sponsor  and  SCAMPI  Lead  Appraiser  have  agreed  to  proceed 
with  appraisal  planning,  based  on  a  common  understanding  of  .the  key 
parameters  that  drive  the  planning  process. 


Documented  agreement(s),  reflected  in  the  appraisal  input,  that  support  a 
common  understanding  of  appraisal  objectives  and  key  appraisal-planning 
parameters. 


1 .2. 1  Tailor  Method 

1.2.2  Identify  Needed  Resources 

1.2.3  Determine  Cost  and  Schedule 

1 .2.4  Plan  and  Manage  Logistics 

1.2.5  Document  and  Manage  Risks 

1.2.6  Obtain  Commitment  to  Appraisal  Plan 


•  Approved  appraisal  plan 

•  Strategy  for  managing  logistics 

•  Strategy  for  preparing  the  organization(s) 

•  Schedule 

•  Interview  plan 

•  Team  assignments 


The  sponsor  and  appraisal  team  leader  agree  on  technical  and  non-technical 
details  for  the  planned  appraisal.  The  plan  is  refined  in  conjunction  with 
performing  the  other  Planning  and  Preparation  phase  activities.  This  agreement 
is  documented  and  reviewed  by  affected  stakeholders  as  appropriate. 


The  final  appraisal  plan  is  reviewed  and  approved. 


Continued  on  next  page 
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1.2  Develop  Appraisal  Plan  (continued) 


Key  Points 


Tools  and 
Techniques 


Metrics 

Verification  and 
Validation 

Records 

Tailoring 


Skilled  appraisal  team  leaders  will  effectively  develop  and  use  outputs  from 
the  other  Planning  and  Preparation  phase  activities  to  achieve  clarity  of  the 
shared  vision  necessary  to  make  the  tradeoffs  and  decisions  resulting  in  a  final 
plan.  This  activity  is  an  important  opportunity  for  the  appraisal  team  leader  to 
demonstrate  process  discipline,  as  well  as  the  type  of  careful  planning 
described  in  the  CMMI  model.  Experienced  appraisal  team  leaders  will 
leverage  data,  templates,  and  assets  (developed  through  their  own  experience) 
to  improve  the  completeness  and  effectiveness  of  the  appraisal  plan, 
recognizing  the  return  on  investment  that  will  be  obtained  through  smooth  and 
efficient  appraisals. 


Tools  include  an  appraisal  plan  template,  samples,  and  embedded  procedural 
guidance  in  planning  templates.  Estimation  worksheets  and  methods  for 
assessing  the  impact  of  appraisal  constraints  are  also  quite  useful. 


•  Calendar  time  spanned  by  the  activity 

•  Effort  consumed  in  carrying  out  the  activities  of  this  process 

•  Level  and  frequency  of  changes  to  the  appraisal  plan 


•  Comparison  of  actual  effort  for  this  activity  with  historical  data 
accumulated  by  the  appraisal  team  leader 

•  Review  of  the  appraisal  plan  by  affected  stakeholders 

•  Sponsor’s  approval  of  the  plan 


•  Estimation  worksheets  (if  used) 

•  Appraisal  plan  (see  activity  1.2.6  for  a  detailed  list  of  plan  contents) 


•  In  some  applications,  planning  templates  and  procedures  in  routine  use 
within  the  organization  can  be  adapted  to  the  needs  of  the  appraisal.  This 
aids  in  communication  as  well  as  local  ownership  of  the  process. 

•  A  structured  planning  workshop  may  be  of  benefit  for  organizations  with 
limited  appraisal  experience.  Such  a  workshop  is  a  valuable  opportunity  to 
discover  risks  as  well  as  mitigation  strategies. 


Continued  on  next  page 


Page  11-40 


CMU/SEI-2002-HB-002 


1 .2  Develop  Appraisal  Plan  (continued) 


interfaces 
with  Other 
Processes 


Summary  of 
Activities 


The  appraisal  plan  will  guide  and  define  the  execution  of  the  appraisal  such 
that  it  is  in  concert  with  the  business  needs  and  constraints.  An  initial  plan  can 
be  generated  immediately  following  consultation  with  the  sponsor.  Further 
refinement  is  done  as  detailed  planning  occurs  and  new  information  comes  to 
light  in  executing  appraisal  planning  and  preparation.  A  final  appraisal  plan 
must  be  completed  prior  to  the  completion  of  process  1.5,  Prepare  for 
Collection  of  Objective  Evidence.  Typically,  resources,  method  tailoring, 
model-related  decisions,  and  a  planned  list  of  outputs  are  finalized  early  on, 
while  cost,  schedule,  and  logistics  are  finalized  later  in  the  Plan  and  Prepare 
for  Appraisal  phase. 

The  appraisal  input  is  a  necessary  input  to  the  appraisal-planning  process. 
While  it  may  not  be  necessary  to  formally  separate  the  requirements  analysis 
activities  from  the  activities  described  in  this  section,  prior  understanding  of 
the  appraisal  requirements  is  a  necessary  input  to  this  process.  The  plan  for  the 
appraisal  provides  an  important  vehicle  for 

•  documenting  agreements  and  assumptions 

•  establishing  and  maintaining  sponsorship 

•  tracking  and  reporting  the  performance  of  the  appraisal  process 

•  reinforcing  commitments  at  key  points  in  the  appraisal  process 

The  distinction  between  the  appraisal  input  and  appraisal  plan  is  intended  to 
separate  the  key  appraisal  requirements  and  strategic  objectives,  which  require 
high  sponsor  visibility  and  change  control  approval,  from  the  tactical  planning 
details  necessary  to  implement  and  satisfy  these  objectives.  While  sponsor 
visibility  into  the  appraisal  plan  is  necessary,  revisions  are  typically  low-level 
implementation  details  and  do  not  ordinarily  require  sponsor  re-approval.  In 
practical  use,  the  appraisal  input  is  often  packaged  as  a  component  of  the 
appraisal  plan,  and  a  single  sponsor  signature  can  serve  as  approval  for  both. 


This  process  is  composed  of  six  activities  summarized  here  and  described 
below.  The  scope  of  the  appraisal  is  defined  in  terms  of  (a)  the  portion  of  the 
CMMI  model  that  will  be  investigated  and  (b)  the  bounds  of  the 
organizational  unit  for  which  the  results  can  be  considered  valid  (e.g.,  a 
project,  a  product  line,  an  operating  division,  a  business  unit,  an  entire  global 
enterprise).  Method-tailoring  choices  are  made  to  most  effectively  achieve 
appraisal  objectives  within  defined  constraints  of  time,  effort,  and  cost.  The 
resources  required  to  carry  out  the  appraisal  are  identified.  The  cost  and 
schedule  are  negotiated  and  recorded.  The  details  of  logistics,  particularly  for 
the  on-site  period,  are  documented.  Risks  and  risk-mitigation  plans  are 
identified  and  documented.  Completion  of  these  activities  results  in  a  well- 
defined,  achievable  appraisal  plan. 
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1 .2  Develop  Appraisal  Plan 


GSS  and  CPM  Implementation  Guidance 

When  using  the  MDD  for  Government  source  selection  and  contract  process  monitoring,  the 

following  should  be  considered: 

Guidance  for  Additional  outputs  of  this  process  may  include  the  following: 

GSS  •  Solicitation  Updates 

o  Section  L  and  M  Language  in  RFP  - 

o  updates  to  the  Source  Selection  Plan  and  evaluation  criteria 
o  updates  to  FBOICBD  and  Bidders  Conference  briefings  to  address  use 
of  SCAMPI  in  source  selection  activities 

•  a  strategy  for  providing  feedback  on  findings  through  discussions  or  other 
processes 

•  a  strategy  for  dealing  with  appraisal  findings  post  award 

Guidance  for  Additional  outputs  of  this  process  may  include  the  following: 

CPM  •  a  strategy  for  appraisals  across  the  program  life  cycle 

•  a  description  of  activities  following  appraisal  (prioritization  of  issues, 
development  of  action  plans,  statusing,  etc.) 

•  award  fee  or  contractual  language  implications 
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1.2.1  Tailor  Method 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Tailoring  of  SCAMPI  includes 

•  selection  of  choices  (if  any)  within  the  Required  Practices 

•  setting  parameters  that  are  allowed  to  vary  within  the  Parameters  and 
Limits 

•  inclusion  of  Optional  Practices 

Because  SCAMPI  is  designed  to  apply  to  a  wide  range  of  appraisal 
applications,  the  tailoring  activity  is  one  that  deserves  careful  and  thoughtful 
attention. 

Using  “partially  satisfied”  and  choosing  to  do  the  appraisal  in  “verification”  or 
“discovery”  mode  are  explicit  selectable  tailoring  options.  This  document  is 
designed  to  clearly  indicate  which  aspects  of  the  method  are  required  and 
which  are  tailorable.  The  Parameters  and  Limits  and  Optional  Practices 
sections  of  each  activity  description  provide  discussions  of  tailoring  options, 
in  context. 

In  addition,  the  appraisal  usage  mode  will  determine  some  tailoring  choices. 


•  Review  and  select  tailoring  options  within  the  Required  Practices  in  each 
activity. 

•  Review  and  set  parameters  within  acceptable  limits,  where  variation  is 
expected. 

•  Review  and  select  appropriate  Optional  Practices. 

•  Ensure  that  the  tailoring  decisions  are  self-consistent  and  that  they  are 
appropriate  in  light  of  the  appraisal  objectives  and  constraints. 

•  Document  the  tailoring  decisions  made. 


The  structure  of  the  MDD  clarifies  which  SCAMPI  features  are  required, 
either  as  a  direct  derivative  of  ARC  requirements  or  as  SCAMPI 
requirements.  Parameters  and  Limits  sections  define  the  allowable  variation 
within  these  method  requirements.  Tailoring  guidance  and  Implementation 
Guidance  are  provided  to  assist  with  tuning  the  method  to  fit  sponsor 
objectives  and  appraisal  constraints.  Method  tailoring  and  implementation 
options  must  be  selected  and  implemented  in  a  way  that  does  not  violate 
SCAMPI  Required  Practices. 


Continued  on  next  page 
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1.2.1  Tailor  Method  (continued) 


Optional 

Practices 


Implementation 

Guidance 


Page  11-46 


Provide  the  sponsor  with  more  than  one  candidate  scenario  for  the  appraisal, 
and  help  them  select  among  the  options. 

Alternatively,  the  appraisal  team  leader  may  define  a  tailored  instance  of  the 
method  and  propose  it  to  the  sponsor  for  approval  or  negotiate  some  of  the 
details. 


This  appraisal  method  offers  a  wide  variety  of  choices  that  allow  the  appraisal 
team  leader  and  sponsor  to  select  appraisal  features  that  best  address  appraisal 
and  business  objectives.  The  SCAMPI  Implementation  Model  is  an  asset 
provided  to  Lead  Appraisers  by  the  CMMI  Steward  that  assists  with 
understanding  SCAMPI  tailoring  and  implementation  choices. 

Method  tailoring  is  directly  related  to  the  organizational  scope  and  model 
scope  decisions.  Most  of  the  allowable  tailoring  options  flow  logically  from 
these  decisions  when  taken  in  context  of  the  appraisal  objectives  and 
constraints.  Tailoring  decisions  typically  affect  the  appraisal  risk.  Typical 
tailoring  choices  that  significantly  impact  appraisal  planning  include 

•  CMMI  model  PAs  encompassed  by  the  appraisal 

•  specification  of  the  organizational  unit  to  be  appraised 

•  number,  experience,  skills,  and  affiliation  (e.g.,  intemal/extemal)  of 
appraisal  team  members 

•  data  collection,  analysis,  and  validation  approaches  to  be  utilized 
including  supporting  work  aids  and  tools 

•  effort  invested  by  the  organization  and  the  appraisal  team  in  preparation, 
including  pre-on-site  data  collection  and  analysis 

•  time  spent  on  site 

Experienced  appraisal  team  leaders  will  provide  a  well-defined  approach  to 
ensure  that  the  appraisal  objectives  are  achieved  in  an  efficient  and  effective 
manner.  Experienced  sponsors  will  require  a  well-defined  approach  to  ensure 
an  acceptable  level  of  risk  in  meeting  objectives  within  the  constraints.  The 
appraisal  plan  documents  the  method-tailoring  decisions  and  their  rationale, 
and  the  associated  method  variations  and  techniques  that  will  be  employed. 
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1.2.1  Tailor  Method 


GSS  and  CPM  Implementation  Guidance 

When  using  the  MDD  for  Government  source  selection  and  contract  process  monitoring,  the 
following  should  be  considered: 

Guidance  for  To  meet  the  appraisal  objectives  for  source  selection  or  contract  process 
GSS  and  CPM  monitoring  activities,  it  may  become  necessary  to  consciously  choose  to  not 
implement  a  required  practice  and  therefore  rely  on  . an  ARC  Class  B  or  Class 
C  appraisal  method  to  gather  the  required  information  to  support  specific 
sponsor  needs. 

Examples  of  specific  appraisal  issues  that  might  be  better  addressed  by  ARC 
Class  B  or  Class  C  appraisal  methods  include 

•  no  requirement  for  goal/maturity/capability  ratings;  only  interested  in 
process  capability  strengths  and  weaknesses 

•  schedule  constraints  that  limit  time  on  site,  leading  to  significant  tailoring 
of  required  practices 

•  source  selection  sensitivities  restricting  instruments  that  can  be  used  to 
validate  preliminary  findings  by  appraised  organization 

•  source  selection  sensitivities  restricting  disclosure  of  final  findings  to 
appraised  organization  until  discussions  phase  of  acquisition 

•  sponsor  goal  for  discovery-based  appraisal  with  higher  percentage  of 
direct  affirmations  and  direct  artifacts  of  process  implementation  and 
adherence,  and  less  reliance  on  indirect  artifacts 


Guidance  for  In  addition  to  the  tailoring  choices  mentioned  above,  sponsors  and  Lead 
GSS  Appraisers  supporting  source  selections  may  need  to  address  additional 

tailoring  decisions: 

•  a  strategy  for  dealing  with  complex  teams  of  offerors  and  with  a  variety  of 
proposed  management  approaches  for  teaming  (ranging  from  typical 
prime/sub  teaming,  where  teammates  use  their  own  corporate  practices,  to 
more  integrated  teaming,  where  new  project  processes  are  defined 
drawing  on  various  teammates’  strengths  and  their  roles  on  the  project) 

o  What  is  the  scope  of  the  organization  “unit”  to  be  appraised  given 
various  teaming  constructs? 
o  Which  teammates  will  be  subject  to  appraisal? 

•  use  of  Contracting  Officers  as  observers  to  accompany  the  appraisal  team 
during  the  detailed  planning,  as  well  as  on-site  appraisal  activities  to 
represent  source  selection  interests 

Any  observers  who  will  be  accompanying  the  appraisal  team  should 
receive  the  same  orientation  as  the  rest  of  the  team.  Observers  should  also 
be  included  in  any  model  and  appraisal-method  training  the  team  receives 
to  gain  a  better  understanding  of  what  they  are  observing.  (See  activities 
1.1.2,  Determine  Appraisal  Constraints,  and  1.3.3,  Prepare  Team.) 


Continued  on  next  page 
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1.2.1  Tailor  Method 


GSS  and  CPM  Implementation  Guidance  (continued) 


Guic:»cefor 
GSS,  cont. 


•  whether  the  source  selection  schedule  can  be  supported  with  a  single  team 
conducting  appraisals  across  all  offerors  or  if  parallel  teams  will 
simultaneously  conduct  appraisals  to  compress  the  source  selection 
schedule.  Each  approach  has  distinct  advantages  and  disadvantages.  The 
primary  advantages  to  having  a  single  team  conduct  appraisals  for  all 
offerors  are  consistency  of  treatment  across  the  acquisition  and  a 
consistent  set  of  appraisal  results  to  be  considered  by  the  source  selection 
evaluation  team — one  yardstick  by  which  all  offerors  are  judged.  The 
primary  disadvantage  to  having  a  single  appraisal  team  is  the  time  it  takes 
to  conduct  all  appraisals  in  a  serial  fashion,  rather  than  in  parallel. 

•  the  relationship  of  appraisal  team  members  to  the  source  selection 
evaluation  team  (evaluators  for  specific  evaluation  criteria,  advisors  to  the 
source  selection  team  who  are  independent  of  the  source  selection  team, 
etc.) 

•  pre-proposal  submissions  required  to  support  appraisal  planning  and 
gathering  objective  information 

o  offeror  submission  of  candidate  projects  to  be  appraised 
o  offeror  submission  of  and/or  use  of  instruments  (maturity 
questionnaire,  practice  implementation  indicators) 
o  offeror  submission  of  objective  information  in  support  of  on-site 
appraisal  activities 

•  techniques  for  selecting  projects  to  evaluate  from  candidate  projects 
submitted  by  the  offeror  (criteria  to  be  based  on  relevant  size,  relevant 
domain,  current  life-cycle  phase,  etc.) 

•  techniques  for  communicating  preliminary  findings  and  final  findings  to 
the  offeror  given  the  “discussions”  sensitivities  in  source  selections  (via 
formal  preliminary  findings/final  findings  briefings  during  on-site 
appraisal  activities,  use  of  written  notice  of  “outstanding  information 
needs”  prior  to  conclusion  of  on-site  appraisal,  disclosure  of  final  findings 
during  discussions  phase  of  source  selection,  etc.).  Refer  to  activities  2.2.3 
and  3.1.1,  respectively,  for  more  information  on  techniques  for 
communicating  preliminary  and  final  findings. 

•  use  of  appraisal  data/findings  in  the  source  selection  evaluation  process 
(full  appraisal  report  input  as  an  independent  data  source  into  the  source 
selection  evaluation  process,  pass/fail  criteria  for  offeror  responsiveness 
determination,  etc.) 

The  strategies  for  conducting  the  appraisal  and  for  using  the  appraisal  results 
in  support  of  the  source  selection  decision  should  be  documented  in  the  RFP 
and  source  selection  plan,  as  appropriate.  Sample  paragraphs  are  included  in 
Appendix  F. 
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1.2.1  Tailor  Method 


GSS  and  CPM  Implementation  Guidance  (continued) 

Guidance  for  Additional  tailoring  considerations  may  include 

CPM  •  a  strategy  for  dealing  with  complex  development  teams  and  for  dealing 

with  a  variety  of  management  approaches  for  teaming  (ranging  from 
typical  prime/sub  teaming,  where  teammates  use  their  own  corporate 
practices,  to  more  integrated  teaming,  where  new  project  processes  are 
defined  drawing  on  various  teammates’  strengths  and  their  roles  on  the 
project) 

•  inclusion  of  contractor  representatives  on  the  appraisal  team  or  as 
observers  to  facilitate  site/contractor  understanding  of  appraisal  findings 
and  to  assist  in  post-appraisal  action-plan  generation  to  address  findings 
and  also  to  assist  in  the  implementation  and  monitoring  of  these  process 
improvement  activities 

•  joint  or  parallel  contractor/Govemment  appraisals  to  efficiently  satisfy 
both  corporate  and  Government  process  improvement  objectives 

•  use  of  appraisal  data/findings  by  both  contractor  and  Government 
program  management  and  in  process  improvement  activities 
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1 .2.2  Identify  Needed  Resources 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Optional 

Practices 


This  activity  is  concerned  with  the  identification  and  estimation  of  resources 
needed  to  carry  out  the  appraisal.  Resources  include  personnel,  facilities, 
tools,  and  access  to  information. 


•  Identify  appraisal  team  members. 

•  Identify  appraisal  participants. 

•  Identify  equipment  and  facilities. 

•  Identify  other  appraisal  resources  needed. 

•  Document  resource  decisions  in  the  appraisal  plan. 


The  level  of  detail  in  the  identification  of  needed  resources  must  be  sufficient 
to  support  the  creation  of  the  appraisal  plan.  For  example,  the  appraisal  team 
leader  must  identify 

•  the  names  of  people  who  are  candidates  for  interviews  or  appraisal  team 
membership 

•  the  organizational  or  project  affiliation  of  these  people 

•  the  location,  seating  capacity,  and  configuration  of  rooms  to  be  used  by 
the  team 

•  specific  equipment  needed  (e.g.,  overhead  projector,  laptop  projector, 
video-conferencing) 


Several  months  before  the  appraisal,  tour  the  facility  where  the  appraisal  will 
be  held.  Assign  an  individual  from  the  appraised  organization  to  carry  out  the 
duties  of  the  Organizational  Unit  Coordinator  (administrative  and  logistical 
support;  see  activity  1.3.2.). 


Continued  on  next  page 
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1 .2.2  Identify  Needed  Resources  (continued) 


Implementation 

Guidance 


Appraisal  resources  are  typically  defined  early  in  the  appraisal-planning 
process.  Identifying  resources  goes  hand  in  hand  with  estimating  appraisal 
cost  and  schedule  (see  activity  1.2.3),  and  these  may  be  iteratively  refined. 
Tradeoffs  are  routinely  made  in  light  of  the  appraisal  objectives  and 
constraints. 

The  appraisal  sponsor  or  senior  site  manager  may  identify  candidate  appraisal 
team  members  and  appraisal  participants.  Review  of  the  organizational  unit 
structure  or  other  site-specific  information  can  also  be  useful  for  this.  Initially, 
participants  can  be  specified  in  terms  of  roles  or  responsibilities,  with  specific 
names  to  be  determined  later.  Process  1.3  contains  additional  guidance  on 
selecting  appraisal  team  members. 

Equipment  and  facilities  are  often  negotiated  with  the  organizational  unit 
where  the  appraisal  on-site  activities  will  be  performed,  but  sometimes  these 
must  be  acquired.  A  room  for  dedicated  use  by  the  appraisal  team  is  usually 
necessary  for  private  discussions  and  to  protect  the  confidentiality  of  appraisal 
data.  Ideally,  this  is  separate  from  the  other  rooms  where  interview  sessions 
are  held. 

The  availability  of  computing  resources,  such  as  computers,  printers,  and 
networks,  is  a  key  consideration  that  should  be  planned  and  understood. 
Access  to  special  tools  or  applications  may  also  be  needed. 
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1 .2.2  Identify  Needed  Resources 


GSS  and  CPM  Implementation  Guidance 

When  using  the  MDD  for  Government  source  selection  and  contract  process  monitoring,  the 
following  should  be  considered: 

Guidance  for  To  address  specific  resource  requirements  for  source  selection  and  contract 
GSS  and  CPM  process  monitoring,  the  appraisal  team  leader  should  also 

•  program/budget  for  current  year/future  year  funding  for  training, 
coaching,  and/or  consultant  appraisal  support 

•  select  relevant  location(s)  for  appraisal  activities  (will  various  teammates 
travel  to  one  central  location,  will  selected  engineering  teammates  be 
appraised  at  their  sites,  etc.) 

•  identify  security  clearance  requirements  or  conflict  of  interest  restrictions 
for  appraisal  team  members 

Guidance  for  Since  appraisal  planning  is  an  iterative  process,  the  appraisal  team  lead  should 
GSS  identify  the  types  of  people  who  may  be  interviewed  in  the  early  stages  of 

planning  (pre-solicitation),  and  then  generate  the  list  of  specific  interviewees 
in  later  iterations  of  the  planning  process.  The  organizational  or  project 
affiliation  of  the  interviewees  should  similarly  be  refined  in  an  iterative 
fashion,  with  early  appraisal  plans  identifying  prime/sub  contractor  affiliation 
only,  then  with  specific  project  affiliations  being  identified  in  subsequent 
iterations,  after  projects  to  be  appraised  have  been  selected  from  the  offeror- 
proposed  candidate  list.  Other  considerations  for  resources  may  include 

•  how  the  sponsor’s/acquirer’s  source  selection  sensitive  data  will  be 
protected  during  on-site  appraisal  (safes,  locks,  etc.) 

•  the  number  of  appraisal  teams  and  the  number  of  “spare”  fully  trained 
appraisal  team  members  that  may  be  available  in  case  of  sickness  or  lack 
of  availability  of  primary  team  members 

•  conflict  of  interest  considerations  for  appraisal  team  members 

•  special  considerations  for  potential  interviewees  who  reside  off-site 
(travel,  teleconferencing,  etc.) 

Optional  Practices 

It  may  not  be  feasible  to  tour  the  facility  prior  to  on-site  appraisal  activities. 
Work  with  the  Organizational  Unit  Coordinator  to  ensure  that  facility 
requirements  are  adequately  addressed  for  an  effective  appraisal. 

Guidance  for  Providing  the  appraisal  team  with  direct  access  to  the  contractor’s 

CPM  development  environment  will  reduce  the  preparation  costs  to  the  unit  being 

appraised  (e.g.,  less  need  to  produce  paper  copies  of  project  documentation 
and  artifacts).  Providing  a  robust  set  of  objective  evidence  to  appraisal  team 
members  will  enhance  their  ability  to  understand  the  development 
environment  and  enable  them  to  appropriately  judge  reference  model  practices 
relative  to  the  offeror’s  environment. 
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1 .2.3  Determine  Cost  and  Schedule 


Activity 

Description 

Required 

Practices 


Parameters 
and  Limits 


Optional 

Practices 


A  top-level  cost  breakdown  and  schedule  are  developed  and  included  in  the 
plan. 


•  Estimate  the  duration  of  key  events  as  a  basis  for  deriving  a 
comprehensive  schedule. 

•  Estimate  the  effort  required  for  the  people  participating  in  the  appraisal. 

•  Estimate  the  costs  associated  with  using  facilities  and  equipment  (as 
appropriate). 

•  Estimate  the  costs  for  incidentals  (e.g.,  travel,  lodging,  meals)  as 
appropriate. 

•  Document  detailed  schedule  estimates  in  the  appraisal  plan. 

•  Document  detailed  cost  estimates  in  the  appraisal  plan. 


Effort  estimates  should  be  developed  not  only  for  the  appraisal  team,  but  also 
for  the  expected  participants  within  the  organizational  unit  (e.g.,  interviewees, 
respondents  to  instruments  administered,  attendees  at  briefings,  support  staff). 

Scheduling  for  each  day  of  the  appraisal  is  required. 


None. 


Continued  on  next  page 
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1 .2.3  Determine  Cost  and  Schedule  (continued) 


Implementation 

Guidance 


Cost  and  schedule  may  be  developed  top  down  based  upon  sponsor  objectives 
and  constraints,  bottom  up  based  upon  results  of  other  planning  and 
preparation  processes  and  activities,  or  more  generally  using  a  combination  of 
the  two  approaches.  Scheduling  the  events  and  activities  of  the  appraisal  is  an 
ongoing  logistical  task  that  requires  the  coordination  of  many  different  groups 
of  individuals.  Determining  and  communicating  a  schedule  for  the  appraisal, 
and  maintaining  ongoing  visibility  as  the  details  take  form,  is  the  primary 
responsibility  of  the  appraisal  team  leader.  The  Organizational  Unit 
Coordinator  is  expected  to  provide  support  in  this  task,  and  the  appraisal  team 
leader  typically  selects  the  person  who  plays  that  role  with  this  duty  in  mind. 

The  needs  of  the  sponsor  for  appraisal  outputs  of  a  specified  quality  fulfilling 
a  specified  purpose,  balanced  against  the  resources  available  to  conduct  the 
appraisal,  will  determine  the  schedule  constraints.  Schedule  and  cost  need  to 
be  considered  for  the  entire  span  of  the  appraisal  activities.  The  tradeoff 
between  time  spent  in  preparation  versus  time  spent  on  site  will  therefore  be  a 
significant  factor,  as  will  post-on-site  reporting  activities. 

Organizational  costs  for  preparing  and  supporting  appraisals  can  be  reduced 
by  gathering  and  maintaining  objective  evidence  for  each  project  instance.  In 
addition  to  providing  an  effective  mechanism  for  monitoring  the  process 
implementation  and  improvement  progress  of  each  project,  this  enables  the 
ready  availability  and  reuse  of  objective  evidence  for  subsequent  appraisals. 

While  the  schedule  for  the  appraisal  will  be  shared  with  a  fairly  wide 
audience,  the  cost  of  the  appraisal  (or  elements  within  the  appraisal)  is  often 
kept  from  wide  view,  due  to  the  potentially  sensitive  nature  of  this 
information. 
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1 .2.3  Determine  Cost  and  Schedule 


GSS  and  CPM  Implementation  Guidance 

When  using  the  MDD  for  Government  source  selection  and  contract  process  monitoring,  the 
following  should  be  considered: 

Guidance  for  The  appraisal  plan  should  address  the  appraisal  activities  with  respect  to  the 

GSS  balance  of  the  source  selection  activities.  The  sponsor  and  appraisal  team  lead 

may  need  to  consider 

•  whether  the  appraisals  will  be  conducted  (a)  prior  to  proposal  submission 
(and  prior  to  the  start  of  actual  source  selection  evaluation  activities),  (b) 
after  submission  of  proposals  by  all  offerors,  or  (c)  after  submission  of  the 
technical  proposal  but  prior  to  submission  of  cost  proposals,  as  in  a  split 
proposal  process  (see  Appendix  E  for  more  information  on  the  use  of 
SCAMPI  in  the  split  proposal  process).  The  main  distinction  between 
these  three  options  will  be  the  time  available  in  the  source  selection 
schedule  in  which  appraisals  can  be  conducted. 

•  best-  and  worst-case  schedules,  given  an  estimated  number  of  offeror 
teams  (and  complexity  of  those  teams  in  terms  of  prime/subcontractors). 
The  source  selection  schedule  and  the  appraisal  schedule  could  be  directly 
related  to  both  factors. 

It  may  not  be  appropriate  or  relevant  to  estimate  costs  for  offerors  but  only  for 
the  appraising  entity.  However,  the  strategy  for  conducting  the  appraisals 
should  support  and  be  reconciled  with  the  overarching  schedule  of  the  source 
selection. 


The  appraisal  plan  should  address  the  appraisal  activities  with  program 
development  schedules  and  activities.  The  sponsor  and  appraisal  team  lead 
may  need  to  consider 

•  the  optimum  time,  post  contract  award,  for  conducting  an  appraisal.  Given 
the  size  of  the  program  and  the  engineering  schedule,  the  appraisal  will  be 
most  effective  when  initial  start-up  activities  have  concluded  and  the 
project  team  has  fully  engaged  in  at  least  the  primary  engineering  and 
development  activities  to  support  the  collection  of  relevant  artifacts  and 
affirmations.  In  addition,  the  appraisal  should  be  scheduled  at  a  time  that 
minimizes  conflicts  with  critical  engineering  milestones  or  activities.  If  a 
project  is  using  an  incremental  or  iterative  approach  to  development,  the 
appraisal  may  be  most  supportable  at  the  end  of  an  engineering  cycle.  It 
may  also  be  highly  desirable  to  schedule  the  appraisal  in  conjunction  with 
similar  internal  corporate  appraisal  activities  to  minimize  the  disruption  to 
project  personnel. 


Guidance  for 
CPM 
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1 .2.3  Determine  Cost  and  Schedule 


GSS  and  CPM  Implementation  Guidance  (continued) 

Guidance  for  •  the  time  intervals  between  contract  process  monitoring  appraisals,  to 
CPM,  cont.  identify  whether  process  weaknesses  have  been  addressed  or  whether 

process  strengths  and  capability  maturity  have  been  maintained  or  have 
degraded 

•  the  focus  for  the  subsequent  contract  process  monitoring  appraisals 
(whether  they  will  be  primarily  focused  on  weaknesses  previously 
identified  or  be  more  comprehensive  appraisals  across  the  program’s 
engineering  and  management  activities) 
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1 .2.4  Plan  and  Manage  Logistics 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Optional 

Practices 

Implementation 

Guidance 


The  logistical  details  of  the  on-site  portion  of  the  appraisal  are  negotiated  and 
documented.  The  appraisal  team  leader,  supported  by  the  Organizational  Unit 
Coordinator,  manages  planning  tasks  that  document  and  communicate 
logistical  arrangements.  Checklists  and  action  item  tracking  mechanisms  are 
very  important  structures  used  to  manage  these  tasks. 


•  Document  logistical  schedules  and  dependencies. 

•  Maintain  communication  channels  for  providing  status. 

•  Assign  responsibilities  for  tracking  logistical  issues. 


Effective  planning  depends  on  anticipating  a  variety  of  logistical  issues  that 
may  occur  during  the  appraisal.  Issues  that  are  sometimes  overlooked  include 

•  identifying  hotels  for  people  traveling  to  the  appraisal 

•  providing  workstation  support 

•  ordering  meals 

•  interacting  with  facilities  staff  on  site 

•  meeting  security/classification  requirements 

•  providing  badges  or  arranging  for  escorts  in  limited-access  facilities 


None. 


Every  experienced  appraisal  team  leader  knows  the  value  of  thorough 
logistical  planning  and  tracking.  The  time-critical  nature  of  on-site  appraisal 
activities  makes  it  very  difficult  to  manage  last-minute  changes  in  important 
details  such  as  the  following: 

•  availability  of  conference  rooms  and  meeting  rooms  of  the  appropriate 
size 

•  access  to  rooms,  equipment,  and  supplies  needed  for  administrative  tasks 

•  transportation  and/or  lodging  for  team  members  or  the  remote  members  of 
the  organizational  unit 

•  food  and  other  amenities  required  for  adequate  working  conditions 

•  communication  channels  and  back-up  staff  to  support  the  team  on  site 


♦ 


CMU/SEI-2002-HB-002 


Page  11-59 


1 .2.4  Plan  and  Manage  Logistics 


GSS  and  CPM  implementation  Guidance 

When  using  the  MDD  for  Government  source  selection  and  contract  process  monitoring,  the 
following  should  be  considered: 

Guidance  for  Stipulate  high-level  logistical  requirements  (e.g.,  single  versus  multiple  site 
GSS  visits  for  offerors  composed  of  multiple  bidding  contractors,  duration  of  on¬ 

site  appraisal  activities,  type  of  office  space  required  to  support  the  appraisal 
activities)  in  the  RFP,  along  with  a  description  of  how  logistical  details  will  be 
resolved  with  the  Organizational  Unit  Coordinator  in  advance  of  on-site 
appraisal  activities.  Then  iteratively  coordinate  logistical  details  in  accordance 
with  the  source  selection  schedule. 

Additional  issues  for  consideration  include 

•  interacting  with  facilities  staff  on  site  while  accommodating  Contracting 
Officer  sensitivities  on  what  constitutes  “discussions”  (having  a 
contracting  office  representative  participate  in/facilitate  the  logistical 
coordination  with  the  offeror’s  staff  while  the  appraisal  team  is  on  site, 
etc.) 

•  protection  and  proper  destruction  of  acquisition-sensitive  data,  artifacts, 
and  appraisal  working  products  (appraiser  notes,  etc.) 


Guidance  for  No  additional  guidance. 

CPM 


♦ 


CMU/SEI-2002-HB-002 


Page  11-61 


1 .2.5  Document  and  Manage  Risks 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Optional 

Practices 

Implementation 

Guidance 


As  with  any  project  containing  dependencies  among  events,  people,  and  other 
resources,  risk  management  is  an  important  ingredient  to  success.  The 
appraisal  team  leader  is  responsible  for  documenting  and  communicating  risks 
and  associated  mitigation  plans  to  the  sponsor  and  appraisal  team  members. 


•  Identify  appraisal  risks. 

•  Develop  mitigation  plans  for  key  appraisal  risks,  and  implement  these 
plans  as  necessary. 

•  Keep  the  appraisal  sponsor  and  other  stakeholders  informed  of  the 
appraisal  risk  status. 


The  risks  and  mitigation  plans  identified  through  conducting  this  activity  are 
required  elements  of  the  appraisal  plan  (see  Parameters  and  Limits  for  activity 
1.2.6).  Most  Lead  Appraisers  include  a  section  titled  “Risk  Management”  in 
the  appraisal  plan.  The  level  of  effort  devoted  to  risk-management  activities  is 
something  the  appraisal  team  leader  must  adjust  to  fit  the  situation  at  hand. 


None. 


The  appraisal  plan  is  used  to  document  and  track  risks  to  the  successful 
conduct  of  the  appraisal.  As  with  the  requirement  to  address  logistical  issues 
during  planning,  there  are  no  minimum  guidelines  to  be  met  other  than  the 
requirement  that  the  plan  include  identified  risks  and  planned  mitigation 
strategies. 

The  appraisal  team  leader  is  responsible  for  keeping  the  appraisal  sponsor 
informed  of  risk-management  activities  so  that,  if  needed,  timely  sponsor 
intervention  is  possible  to  ensure  the  achievement  of  appraisal  objectives. 


♦ 
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1 .2.5  Document  and  Manage  Risks 


GSS  and  CPM  Implementation  Guidance 

When  using  the  MDD  for  Government  source  selection  and  contract  process  monitoring,  the 

following  should  be  considered: 

Guidance  for  Typical  risks  for  appraisals  include  the  following: 

GSS  and  CPM  •  incapacitation  of  appraisal  team  members.  Techniques  for  mitigating  this 
risk  could  include  the  identification  and  training  of  alternate  team 
members  who  could  augment  the  team  if  needed,  or  a  plan  for  conducting 
the  appraisal  with  less  than  a  full  appraisal  team.  (Note:  The  SCAMPI 
method  requires  a  minimum  team  of  four  qualified  personnel.  See  activity 
1.3.2,  Select  and  Prepare  Team.) 

•  lack  of  preparation  by  the  appraised  organization  or  offeror  causing  the 
appraisal  team  to  resort  to  a  more  time-consuming  discovery-mode 
appraisal.  A  possible  mitigation  technique  would  be  to  prioritize  the  PAs 
as  to  primary  or  secondary  (see  discussion  in  activity  1.1.3,  Determine 
Appraisal  Scope).  Primary  PAs  would  have  to  be  fully  addressed  by  the 
appraisal  team,  while  secondary  PAs  would  be  addressed  if  time  permitted 
and  would  possibly  not  be  rated.  A  second  mitigation  technique  to  deal 
with  the  risk  of  an  offeror  not  being  adequately  prepared  to  support  a 
verification-based  appraisal  for  GSS  could  be  to  require,  in  the  RFP,  that 
all  offerors  totally  complete  the  Practice  Implementation  Indicator 
Descriptions  (PIIDs)  for  some  number  of  projects  (possibly  after  the 
Government  has  selected  the  projects).  That  way,  if  the  PIIDs  are  not 
filled  out,  the  offeror  could  be  judged  to  be  not  responsive  to  the  RFP. 
(See  activity  1.4.3,  Obtain  Initial  Objective  Evidence.) 


♦ 
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1 .2.6  Obtain  Commitment  to  Appraisal  Plan 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Formal  sponsor  commitment  is  obtained  to  the  appraisal  plan.  The  appraisal 
plan  constitutes  a  “contract”  between  the  appraisal  sponsor  and  the  appraisal 
team  leader,  so  it  is  vital  that  this  agreement  be  formal. 


•  Document  the  appraisal  plan. 

•  Review  the  appraisal  plan  with  the  sponsor  and  secure  the  sponsor’s 
approval. 

•  Provide  the  appraisal  plan  to  relevant  stakeholders  for  review. 


Required  contents  of  the  appraisal  plan  include  the  following,  at  a  minimum: 

•  the  appraisal  input  (see  activity  1.1.5) 

•  the  activities  to  be  performed  in  conducting  the  appraisal 

•  resources  needed  for  conducting  the  appraisal  (see  1.2.2) 

•  cost  and  schedule  estimates  for  performing  the  appraisal  (see  activity 
1.2.3) 

•  appraisal  logistics  (see  activity  1.2.4) 

•  risks  and  mitigation  plans  associated  with  appraisal  execution  (see  activity 
1.2.5) 

•  the  criteria  to  verify  that  the  requirements  of  ISO/IEC  15504  have  been 
met,  if  requested  by  the  appraisal  sponsor 

There  must  be  a  signature  block  for  the  appraisal  team  leader  and  the  sponsor 
to  indicate  in  writing  their  commitment  to  the  plan.  If  minor  updates  are  made 
to  the  plan,  signatures  do  not  have  to  be  obtained  again  except  when  one  or 
more  elements  of  the  appraisal  input  have  been  changed. 

At  a  minimum,  the  appraisal  team  members  are  considered  relevant 
stakeholders  and  should  receive  a  copy  of  the  approved  appraisal  plan. 


Continued  on  next  page 
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1 .2.6  Obtain  Commitment  to  Appraisal  Plan  (continued) 


Optional 

Practices 

Implementation 

Guidance 


Page  11-68 


Use  a  signature  block  for  relevant  stakeholders  to  indicate  in  writing  their 
commitment  to  the  plan  (i.e.,  each  team  member  signs  the  plan). 


While  sponsor  visibility  into  the  appraisal  plan  is  necessary,  revisions  are 
typically  low-level  implementation  details  and  do  not  ordinarily  require 
sponsor  re-approval.  This  is  in  contrast  to  the  appraisal  input,  which  contains 
strategic,  key  appraisal  requirements,  objectives,  and  constraints.  Revisions  to 
the  appraisal  input  must  be  approved  by  the  sponsor.  In  practical  use,  the 
appraisal  input  is  often  packaged  as  a  component  of  the  appraisal  plan,  and  a 
single  sponsor  signature  can  serve  as  approval  for  both.  The  separation  of  the 
appraisal  input  and  appraisal  plan  is  intended  to  provide  an  appropriate  level 
of  sponsor  visibility  and  approval,  while  leaving  appraisal  team  leaders  the 
flexibility  to  refine  the  low-level  details  necessary  to  complete  thorough 
appraisal  planning. 

The  use  of  the  term  “relevant  stakeholder”  in  the  context  of  appraisal  planning 
is  intended  to  be  interpreted  broadly  to  include  as  many  of  the  participants  and 
other  affected  parties  as  feasible. 


♦ 
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1 .2.6  Obtain  Commitment  to  Appraisal  Plan 


GSS  and  CPM  Implementation  Guidance 

When  using  the  MDD  for  Government  source  selection  and  contract  process  monitoring,  the 
following  should  be  considered: 

Guidance  for  Planning  for  the  appraisals  will  be  an  iterative  process  and  stakeholder 

GSS  agreement  and  commitment  should  be  periodically  renewed.  As  the  appraisal 

plan  matures,  the  appraisal  team  leader  should  ensure  that  the  source- 
selection-relevant  details  of  the  appraisal  plan  remain  consistent  with  the  RFP 
and  source  selection  documentation,  including  the  source  selection  plan.  Non¬ 
source-selection-sensitive  components  of  the  plan  (for  example,  the  site 
visitation  order,  the  on-site  appraisal  schedule  of  activities,  the  list  of  team 
members,  etc.)  should  be  captured  in  the  appraisal  plan  but  should  not  be 
included  in  the  RFP  and  source  selection  planning  documents. 

The  signature  of  the  Contracting  Officer  should  also  be  included. 


Guidance  for  No  additional  guidance. 

CPM 


♦ 


CMU/SEI-2002-HB-002 


Page  11-69 


1.3  Select  and  Prepare  Team 


Purpose 
Entry  Criteria 

Inputs 

Activities 

Outputs 

Outcome 

Exit  Criteria 


Ensure  that  an  experienced,  trained,  appropriately  qualified  team  is  available 
and  prepared  to  execute  the  appraisal  process. 


•  Appraisal  requirements  have  been  documented  (at  least  in  draft  form). 

•  Appraisal  constraints  are  understood  and  documented  (at  least  in  draft 
form). 

•  The  appraisal  plan  is  defined  (at  least  in  draft  form). 


•  Appraisal  requirements  and  constraints  (in  draft  or  final  form) 

•  Appraisal  plan  (in  draft  or  final  form) 

•  Team  training  materials 


1.3.1  Identify  Team  Leader 

1 .3.2  Select  Team  Members 

1.3.3  Prepare  Team 


•  Training  records 

•  Appraisal  team  member  assignments  and  qualifications 

•  A  prepared  appraisal  team  that  has  completed 

appraisal  method  training 
-  reference  model  training 
team-building  activities 
team  orientation  regarding  appraisal 


The  successful  completion  of  this  process  results  in  an  experienced,  trained, 
and  oriented  team  ready  to  execute  the  appraisal.  The  appraisal  team  members 
have  acquired  the  necessary  knowledge  to  play  their  roles,  or  their  previous 
knowledge  is  confirmed  to  be  satisfactory.  The  appraisal  team  leader  has 
provided  opportunities  to  practice  the  skills  needed  for  each  person  to  play  his 
or  her  role,  or  has  confirmed  that  these  skills  have  already  been  demonstrated 
in  the  past.  The  team  members  have  been  introduced  to  one  another,  and  have 
begun  to  plan  how  they  will  work  together. 


•  The  prepared  team  is  committed  to  the  appraisal. 

•  Training  has  been  provided  and  its  results  recorded. 

•  Remediation  of  knowledge/skill  shortfalls  has  been  completed  (if  needed). 


Continued  on  next  page 
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1 .3  Select  and  Prepare  Team  (continued) 


Key  Points 


Tools  and 
Techniques 


Metrics 


Verification  and 
Validation 


Records 


Tailoring 


Whether  the  appraisal  team  leader  trains  an  intact  team  or  forms  a  team  from  a 
corps  of  experienced  team  members,  the  responsibility  to  ensure  that  the  team 
is  ready  to  succeed  rests  with  the  appraisal  team  leader. 


Training  course  material  is  available  from  the  CMMI  Steward  for  training 
teams.  This  should  be  tailored  or  supplemented  by  the  appraisal  team  leader 
based  on  the  appraisal  context  or  degree  of  team  member  experience.  Case 
studies  and  exercises  are  recommended  to  reinforce  the  situations  team 
members  are  likely  to  encounter  during  the  appraisal. 

Other  ways  of  accomplishing  this  activity  may  draw  on  one  or  more  of  the 
following: 

•  providing  supplementary  training  to  previously  experienced  team  members, 
so  that  the  operational  details  of  the  approach  used  will  be  familiar 

•  training  a  cadre  of  team  members  and  keeping  their  knowledge  and  skills 
up-to-date,  as  part  of  an  overall  program  of  appraisals 


•  Summary  of  team  member  qualifications 

•  Effort  and  calendar  time  expended  to  accomplish  training 

•  Trainee  ratings  of  instructional  materials  and  approach  (if  applicable) 

•  Achievement  of  milestones  for  remedial  activities  (if  applicable) 


•  Sponsor  and  appraisal  team  leader  approval  of  team  membership  and 
preparation 

•  Results  of  exams  used  to  demonstrate  training  effectiveness  (if  used) 

•  Feedback  from  team  members  on  their  readiness  to  perform  their  role(s) 


•  Team  member  contact  information 

•  Training  records  (if  applicable) 

•  Feedback  provided  by  trainees  (if  applicable) 

•  Team  qualification  summary  (recorded  in  appraisal  plan) 


•  Case  study  materials  provide  a  variety  of  options  for  expanding  the  team 
training  course  to  add  emphasis  where  more  is  desired. 

•  Experienced  appraisal  team  leaders  have  had  success  conducting  role- 
plays  and  simulated  appraisal  activities  without  case  studies  as  well. 

•  When  assembling  a  team  of  already-trained  members,  it  is  important  to 
conduct  team-building  activities  to  ensure  team  cohesion.  Many  team 
building  exercises  are  available  for  this  purpose 

•  Team  size,  skills,  and  composition  are  tailoring  options  in  the  method. 


Continued  on  next  page 
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1 .3  Select  and  Prepare  Team  (continued) 


Interfaces 
with  Other 
Processes 


Summary  of 
Activities 


This  process  includes  selecting  and  preparing  the  appraisal  team.  It  may  occur 
after  obtaining  sponsor  commitment  to  the  appraisal  input.  The  appraisal  plan 
should  be  available,  at  least  in  draft  form,  as  a  necessary  input  (see  activity 
1.2.6  for  contents).  Selected  appraisal  team  members  may  provide  input  into 
further  definition  of  the  appraisal  planning.  Appraisal  team  training  may 
provide  an  initial  means  to  obtain  a  preliminary  understanding  of  the 
appraised  organization’s  operations  and  processes.  If  available,  the 
organizational  unit’s  PII  database  is  a  useful  resource  for  orienting  the 
appraisal  team  on  organizational  characteristics,  such  as  the  application 
domain,  the  organizational  structure,  the  process  improvement  structure,  and 
approaches  for  reference  model  implementation. 


The  appraisal  team  is  a  cohesive  unit  of  trained  and  capable  professionals, 
each  of  whom  must  meet  stringent  qualifications.  An  appraisal  team  leader  is 
selected  to  plan  and  manage  the  performance  of  the  appraisal,  delegate 
appraisal  tasks  to  team  members,  and  ensure  adherence  to  SCAMPI 
requirements.  Appraisal  team  members  are  selected  based  on  defined  criteria 
for  experience,  knowledge,  and  skills  to  ensure  an  efficient  team  capable  of 
satisfying  the  appraisal  objectives.  Training  is  provided  to  ensure  proficiency 
in  the  reference  model  and  appraisal  method. 


♦ 
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1 .3  Select  and  Prepare  Team 


GSS  and  CPM  Implementation  Guidance 

When  using  the  MDD  for  Government  source  selection  and  contract  process  monitoring,  the 
following  should  be  considered: 

Depending  on  the  number  of  offerors  that  require  a  SCAMPI  appraisal  and  the 
amount  of  time  available  to  perform  the  appraisal  activities,  the  sponsor  may 
choose  to  commission  multiple  SCAMPI  teams  to  perform  appraisals  in 
parallel.  In  cases  where  multiple  teams  are  used,  additional  issues  associated 
with  ensuring  consistency  with  teams,  specialized  domain  knowledge 
requirements,  coordinating  multi-team  activities,  and  the  use  of  results  from 
different  teams  within  the  same  source  selection  activity  should  be  explicitly 
addressed  in  the  appraisal  plan  or  plans.  Regardless  of  the  number  of  teams 
used,  each  team  will  need  to  have  its  own  authorized  Lead  Appraiser,  as  well 
as  a  fully  trained  and  qualified  team. 

Guidance  for  Consider  including  members  from  the  development  organization  for  ongoing 

CPM  contract  process  monitoring  activities  in  which  there  exists  a  long-term 

partnership  between  the  acquiring  organization  and  the  developer. 

♦ 


Guidance  for 
GSS 
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1.3.1  Identify  Team  Leader 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Optional 

Practices 


The  appraisal  sponsor  is  responsible  for  selecting  an  appraisal  team  leader 
who  has  the  appropriate  experience,  knowledge,  and  skills  to  take 
responsibility  for  and  lead  the  appraisal.  By  definition  an  appraisal  team 
leader  must  be  a  SCAMPI  Lead  Appraiser,  authorized  by  the  SEI  Appraiser 
Program,  and  must  be  a  member  of  that  program  in  good  standing.  The  SEI 
Appraiser  Program  is  described  on  the  SEI  Web  site  at 
<http://www.sei.cmu.edu/managing/app.directory.html>.  The  appraisal  team 
leader  is  responsible  for  ensuring  that  the  appraisal  is  conducted  in  accordance 
with  SCAMPI  requirements,  with  tailoring  to  meet  appraisal  objectives  and 
constraints  within  allowable  bounds  defined  by  the  method. 


•  Select  an  authorized  SCAMPI  Lead  Appraiser  to  serve  as  the  appraisal 
team  leader. 

•  Verify  the  qualifications  of  the  appraisal  team  leader  (experience, 
knowledge,  and  skills). 


The  appraisal  team  leader  must  be  an  SEI-authorized  SCAMPI  Lead 
Appraiser  in  good  standing.  This  can  be  verified  on  the  Web  or  by  contacting 
the  SEI  CMMI  Steward  directly. 

There  can  be  only  one  official  appraisal  team  leader  on  any  given  appraisal. 
The  appraisal  team  leader  has  sole  discretion  to  delegate  important  tasks  to 
appraisal  team  members,  but  cannot  delegate  leadership  responsibility  or 
ultimate  responsibility  for  the  successful  completion  of  the  appraisal.  The 
inclusion  of  multiple  Lead  Appraisers  on  a  team  for  a  given  appraisal  can  be  a 
strong  asset  for  the  leader  of  that  team.  However,  the  single  designated 
appraisal  team  leader  must  perform  the  leadership  role  and  manage  the 
appraisal  process. 


In  some  uses  of  SCAMPI,  representatives  of  the  appraisal  sponsor  may 
perform  a  substantial  part  of  the  appraisal  team  leader’s  responsibilities  in 
advance  of  the  initial  identification  of  an  appraisal  team  leader.  Infrastructures 
established  to  manage  Supplier  Selection,  for  example,  may  employ  standard 
acquisition  processes  that  have  well-understood  interfaces  with  the  appraisal 
process. 


Continued  on  next  page 
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1 .3.1  Identify  Team  Leader  (continued) 


Implementation 

Guidance 


SCAMPI  Lead  Appraisers,  by  definition,  will  have  participated  on  a  minimum 
of  three  appraisals  (two  as  an  appraisal  team  member  and  one  as  an  appraisal 
team  leader).  These  requirements  are  outlined  in  the  SEI  Lead  Appraiser 
program.  An  additional  consideration  impacting  team  experience 
requirements,  however,  is  the  appraisal  usage  mode  for  SCAMPI.  Additional 
experience  may  be  necessary  for  the  appraisal  team  leader  and/or  appraisal 
team  members  if  the  appraisal  is  for  Supplier  Selection  and/or  Process 
Monitoring  or  if  it  will  focus  heavily  on  one  of  the  other  available  disciplines 
or  environments,  such  as  acquisition  or  Integrated  Product  and  Process 
Development.  Similarly,  if  the  appraisal  will  be  used  in  a  high  maturity 
organization  (maturity  levels  4-5  or  capability  levels  4-5),  special  experience, 
training,  and/or  expertise  (e.g.,  statistical  process  control)  may  be  necessary 
for  that  specific  appraisal. 

Appraisal  team  leader  responsibilities  are  defined  and  described  throughout 
the  SCAMPI  MDD,  but  a  summary  overview  of  these  responsibilities  includes 
the  following: 

•  Confirm  the  sponsor’s  commitment  to  proceed  with  the  appraisal. 

•  Ensure  that  appraisal  participants  are  briefed  on  the  purpose,  scope,  and 
approach  of  the  appraisal. 

•  Ensure  that  all  appraisal  team  members  have  the  appropriate  experience, 
knowledge,  and  skills  in  the  appraisal  reference  model  and  in  SCAMPI. 

•  Ensure  that  the  appraisal  is  conducted  in  accordance  with  the  documented 
SCAMPI  method. 

•  Verify  and  document  that  the  appraisal  method  requirements  have  been 
met. 

The  appraisal  team  leader  may  be  selected  at  any  time  in  the  appraisal¬ 
planning  phase;  preferably,  the  appraisal  team  leader  is  selected  upon 
initiation  of  appraisal  activities  so  that  he  or  she  may  participate  in  analyzing 
the  requirements  with  the  appraisal  sponsor.  In  any  event,  the  appraisal  team 
leader  should  be  identified  in  time  to  (a)  review  and  approve  the  appraisal 
plan  with  the  appraisal  sponsor  prior  to  beginning  the  on-site  portion  of  the 
appraisal,  and  (b)  ensure  adequate  planning  and  the  preparation  of  appraisal 
team  members. 
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GSS  and  CPM  Implementation  Guidance 

When  using  the  MDD  for  Government  source  selection  and  contract  process  monitoring,  the 
following  should  be  considered: 

Guidance  for  SCAMPI  activities  may  be  part  of  an  overall  plan  for  a  given  source  selection 

GSS  and  CPM  or  contract  monitoring  activity.  The  planning  for  the  overall  activity  of  which 

the  SCAMPI  is  a  part  will  typically  be  performed  and  led  by  the  sponsor  or 
his/her  designated  lead,  who  may  not  be  the  SCAMPI  Lead  Appraiser.  A  clear 
distinction  between  the  source  selection  or  contract  process  monitoring  plan 
and  SCAMPI  activities  should  be  documented  in  the  Appraisal  Plan  and 
should  include  a  description  of  how  the  SCAMPI  Lead  Appraiser  interfaces 
with  the  overall  lead  for  the  source  selection  or  contract  process  monitoring 
activities. 
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1 .3.2  Select  Team  Members 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


This  activity  involves  identifying  available  personnel,  assessing  their 
qualifications,  and  selecting  them  to  become  appraisal  team  members.  It  may 
occur  after  obtaining  the  sponsor’s  commitment  to  conduct  the  appraisal  and 
may  provide  input  to  the  appraisal  planning. 


•  Ensure  that  minimum  criteria  for  individual  team  members  are  met. 

•  Ensure  that  minimum  criteria  for  the  team  as  a  whole  are  met. 

•  Document  the  qualifications  and  responsibilities  of  team  members  in  the 
appraisal  input. 


The  minimum  acceptable  team  size  for  a  SCAMPI  appraisal  is  four  people 
(including  the  team  leader).  The  maximum  recommended  team  size  is  nine. 

All  team  members  must  have  previously  completed  the  SEI-licensed 
Introduction  to  CMMI  course,  delivered  by  an  instructor  who  is  authorized  by 
the  SEI. 

Team  members’  training  in  the  appraisal  method  is  discussed  in  activity  1.3.3, 
Prepare  Team. 

With  regard  to  engineering  field  experience,  the  team  (as  a  group)  must  have 
an  average  of  at  least  6  years  of  experience,  and  the  team  total  must  be  at  least 
25  years  of  experience,  in  each  of  the  disciplines  to  be  covered  in  the 
appraisal. 

With  regard  to  management  experience,  the  team  (as  a  group)  must  have  a 
total  of  at  least  10  years  of  experience,  and  at  least  one  team  member  must 
have  at  least  6  years  of  experience  as  a  manager. 

The  team  should,  in  aggregate,  have  representative  experience  in  the  life 
cycles  in  use  within  the  appraised  organization.  For  any  given  life-cycle 
phase,  at  least  two  members  of  the  team  should  have  experience  as  a 
practitioner. 

Team  members  should  not  be  managers  of  one  of  the  selected  projects  or  be 
within  the  direct  supervisory  chain  of  any  of  the  anticipated  interviewees. 


Continued  on  next  page 


C  MU/SE 1-2002-H  B-002 


Page  11-81 


1 .3.2  Select  Team  Members  (continued) 


Optional  Although  not  required  in  the  Parameters  and  Limits  section  above,  the 

Practices  following  are  considered  recommended  best  practices  and  should  be 

employed  whenever  feasible: 

•  Each  member  should  have  good  written  and  oral  communication  skills, 
the  ability  to  facilitate  the  free  flow  of  communication,  and  the  ability  to 
perform  as  team  players  and  negotiate  consensus. 

•  At  least  half  of  the  team  members  should  have  participated  in  a  previous 
process  appraisal. 

•  Team  members  should  be  perceived  by  the  appraisal  sponsor  as  credible. 

Additional  appraisal  team  member  selection  considerations: 

•  Consider  the  personal  characteristics  of  individual  team  members  (e.g., 
communication  preferences,  personality  types)  and  how  these  may  affect 
the  dynamics  in  a  team  environment. 

•  Use  one  or  more  authorized  Lead  Appraisers  as  team  members. 


Implementation  Appraisal  team  members  are  selected  to  provide  a  diverse  set  of  qualified 
Guidance  professionals  with  the  appropriate  experience,  knowledge,  and  skills  to  make 

reasoned  judgments  regarding  implementation  of  the  reference  model. 

The  accuracy  and  credibility  of  the  appraisal  results  depends  greatly  on  the 
capability,  qualifications,  and  preparation  of  the  appraisal  team  members.  In 
addition  to  the  qualifications  described  above,  other  factors  that  may  affect  the 
performance  of  the  team  or  reliability  of  appraisal  results  should  be 
considered.  Appraisal  constraints,  such  as  security  classification,  may  result  in 
additional  criteria  for  team  member  selection. 

The  selected  appraisal  team  members  and  their  organizational  affiliation  and 
qualifications  (individually  and  in  aggregate)  are  documented  in  the  appraisal 
plan.  Appraisal  team  members  are  typically  selected  from  a  pool  of  qualified 
individuals  provided  by  the  appraisal  sponsor  or  his/her  designee.  The 
appraisal  team  leader  is  the  final  authority  on  acceptance  of  appraisal  team 
members,  and  is  responsible  for  ensuring  their  qualifications  and  suitability 
for  the  appraisal  purpose. 

Situations  where  a  conflict  of  interest  may  arise  should  be  avoided.  Team 
members  who  manage  people  or  processes  in  the  organization  may  struggle 
with  their  ability  to  be  objective.  Team  members  who  are  directly  impacted  by 
the  appraisal  outcome  may  be  distracted  by  the  potential  consequences  of  the 
decisions  they  contribute  to  on  the  appraisal  team. 
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1 .3.2  Select  Team  Members 


GSS  and  CPM  Implementation  Guidance 

When  using  the  MDD  for  Government  source  selection  and  contract  process  monitoring,  the 
following  should  be  considered: 

Guidance  for  For  competitive  source  selection  activities,  acquisition  regulations  may 
GSS  impose  additional  constraints  on  team  membership  in  order  to  ensure  the 

objectivity  of  the  appraisal  team.  For  example,  appraisal  team  members  may 
be  excluded  if  they  have  a  conflict  of  interest  with  any  of  the  offerors. 

Program  requirements  may  drive  selection  of  appraisal  team  members  due  to 
domain  or  technologies  knowledge  or  experience  requirements. 

Guidance  for  No  additional  guidance. 

CPM 

♦ 


CMU/SEI-2002-HB-002 


Page  11-83 


1 .3.3  Prepare  Team 


Activity 

Description 


Required 

Practices 


The  appraisal  team  leader  is  responsible  for  ensuring  that  appraisal  team 
members  are  sufficiently  prepared  for  performing  the  planned  appraisal 
activities.  This  includes  familiarity  with  the  reference  model,  SCAMPI,  the 
appraisal  plan,  organizational  data  and  characteristics,  and  the  tools  and 
techniques  to  be  used  during  the  appraisal.  Roles  and  responsibilities  are 
assigned  for  appraisal  tasks.  Team  building  exercises  are  used  to  practice 
facilitation  skills  and  reach  unity  in  understanding  the  team  objectives  and 
how  they  will  be  satisfied. 

All  team  members  are  expected  to  observe  strict  rules  for  confidentiality,  the 
protection  of  proprietary  or  sensitive  data,  and  the  non-attribution  of 
information  to  project  participants.  Non-disclosure  statements  are  often  used 
to  formalize  these  understandings. 


•  Ensure  that  appraisal  team  members  have  received  reference  model 
training. 

•  Provide  appraisal  method  training  to  appraisal  team  members  or  ensure 
that  they  have  already  received  it. 

•  Provide  for  team  building  and  establishing  team  norms. 

•  Provide  orientation  to  team  members  on  appraisal  objectives,  plans,  and 
their  assigned  roles  and  responsibilities. 


Continued  on  next  page 
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1.3.3  Prepare  Team  (continued) 


Parameters 
and  Limits 


Model  training  must  be  provided  using  the  standard  Introduction  to  CMMI 
course,  delivered  by  an  instructor  who  is  authorized  by  the  CMMI  Steward. 

Method  training  may  be  delivered  in  one  of  two  ways: 

1.  method  training  specific  to  the  appraisal  at  hand 

2.  method  training  delivered  to  a  large  group  of  potential  future  team 
members  who  are  not  currently  engaged  in  an  appraisal 

Method  training  delivered  to  an  intact  team  must  be  at  least  two  days  in 
duration  and  must  emphasize  the  situations  likely  to  be  encountered  by  team 
members  during  the  appraisal.  This  training  will  not  necessarily  cover  all 
variants  in  the  application  of  SCAMPI. 

Method  training  delivered  to  groups  of  potential  future  team  members  must 
cover  the  complete  set  of  tailoring  options  and  allowable  variations  for  the 
method  to  prepare  them  for  a  range  of  situations  they  are  likely  to  encounter 
on  future  appraisals.  The  SEI  Appraiser  Program  specifies  additional 
requirements  about  delivering  training  to  people  who  are  not  already  members 
of  an  appraisal  team. 

Team  members  who  have  previously  been  trained  as  a  member  of  a  prior 
appraisal  team  are  not  automatically  qualified  to  participate  on  a  subsequent 
appraisal  without  first  attending  method  training.  In  such  cases,  the  appraisal 
team  leader  is  required  to  understand  the  nature  of  the  training  delivered 
previously  and  the  adequacy  of  that  training  for  the  appraisal  at  hand.  This 
requires  that  the  previous  appraisal  be  compared  with  the  planned  appraisal. 
For  example,  if  the  team  member  participated  in  an  appraisal  focused  only  on 
software  engineering,  using  the  continuous  representation,  and  the  planned 
appraisal  is  focused  on  SE/SW/IPPD  using  a  staged  representation,  there  may 
be  some  important  new  concepts  to  cover  with  that  team  member. 

There  must  be  at  least  one  event  where  the  team  gathers  as  a  group  for  the 
purpose  of  establishing  team  norms  and  operational  decisions  about  how  the 
team  will  work  for  the  appraisal  at  hand. 
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Optional 

Practices 


Implementation 

Guidance 


Implementation 

Guidance 

Training  in 
the  Reference 
Model 


Some  organizations  have  established  an  “organic”  capability  to  perform 
appraisals  with  very  limited  preparation  effort,  through  the  use  of  a  pool  of 
trained  appraisal  team  members.  Drawing  from  an  established  group  of 
experts,  who  are  accustomed  to  working  together,  clearly  provides  a  savings 
over  time  for  organizations  that  conduct  frequent  appraisals. 


The  team  training  event  is  a  good  place  to  review  the  appraisal  plan  with 
appraisal  team  members,  having  sent  it  to  them  in  advance  of  their  arrival. 
This  event  provides  the  orientation  for  the  entire  appraisal  that  all  appraisal 
team  members  need  to  execute  their  roles  appropriately.  This  also  is  in 
keeping  with  the  “Provide  appraisal  plan  to  relevant  stakeholders  for  review” 
required  practice  in  activity  1.2.6. 

Additionally,  the  team  training  event  is  a  primary  opportunity  to  conduct 
activity  1.5.1,  Perform  Readiness  Review.  The  assembled,  trained  appraisal 
team  can  then  appropriately  assess  the  organization’s  readiness  for  the 
appraisal  and  validate  the  reasonableness  of  appraisal  estimates. 


A  typical  model  training  course  is  delivered  in  two-and-a-half  to  three  days. 
Although  training  in  either  model  representation  (staged  or  continuous)  is 
allowable,  it  is  recommended  that  this  training  be  provided  for  the  model 
representation  to  be  used  during  the  appraisal.  The  successful  completion  of 
reference  model  training  should  precede  training  in  the  appraisal  method. 
There  is  no  “aging”  requirement  for  when  this  model  training  was  received, 
but  the  appraisal  team  leader  is  responsible  for  ensuring  that  each  team 
member  has  adequate  reference  model  understanding,  and  for  taking  remedial 
action  if  necessary.  Attendance  at  model  training  needs  to  be  recorded  by  the 
training  instructor  and  provided  to  the  CMMI  Steward,  in  accordance  with  the 
terms  of  the  instructor  authorization. 

For  appraisals  that  include  higher  levels  (i.e.,  maturity/capability  levels  4  and 
5)  team  members  may  benefit  from  receiving  additional  training  on  this 
subject  matter.  The  Intermediate  Concepts  of  CMMI  course,  a  course  on 
Statistical  Process  Control,  and/or  other  advance  topics  may  be  of  use  for  this 
added  level  of  preparation. 
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1 .3.3  Prepare  Team  (continued) 


Implementation 

Guidance 

Training  in 
the  Appraisal 
Method 


Implementation 

Guidance 

Familiarization 
with  the 
Appraisal  Plan 


A  typical  delivery  of  appraisal  team  training  might  take  two-and-a-half  to 
three  days.  More  or  less  time  may  be  necessary  depending  on  the  relative 
experience  of  the  appraisal  team  members. 

Exercises  in  appraisal  techniques  and  team  development  are  used  to  reinforce 
the  skills  that  will  be  important  during  conduct  of  the  appraisal.  It  is 
recommended  that  exercises  be  used  that  are  appropriate  for  the  organizational 
unit  being  appraised.  Where  sufficient  organizational  artifacts  exist,  “live” 
data  can  be  collected  and  used  in  training  exercises  where  appropriate.  Just-in- 
time  training  can  also  be  used  to  re-emphasize  method  concepts  at  appropriate 
points  in  the  appraisal  process  during  which  the  skills  will  be  utilized. 

Appraisal  team  training  materials  should  be  tailored  to  fit  team  needs  and 
objectives  of  the  specific  appraisal.  Tailoring  provides  opportunities  to 

•  provide  insight  into  the  context,  objectives,  and  plans  of  the  particular 
appraisal 

•  communicate  team  members’  assigned  roles  and  responsibilities 

•  identify  tailoring  of  SCAMPI  for  the  upcoming  appraisal 

•  acquaint  the  team  with  the  organizational  unit’s  characteristics  and 
documentation 

•  focus  on  skills  that  may  be  more  critical  to  the  upcoming  appraisal,  such 
as  the  ability  to  facilitate  interviews  or  the  ability  to  identify  alternative 
practices 

It  is  recommended  that  this  training  be  provided  within  60  days  of  the 
appraisal.  The  appraisal  team  leader  typically  provides  method  training,  but 
other  delivery  options  are  also  acceptable  (as  described  above).  Although 
alternative  training  options  can  provide  some  advantages  and  efficiencies  for 
method  training,  there  are  also  potential  consequences  that  might  be  felt  by 
the  appraisal  team  leader  on  a  given  appraisal,  such  as  poor  training  quality  or 
readiness  of  team  members.  Regardless  of  how  method  training  is  delivered  to 
the  team  members,  opportunities  for  team  building  should  be  provided  to 
coalesce  the  team  and  bring  the  team  up  to  speed  on  the  specifics  of  the 
appraisal  being  planned. 


Method  training  and  team  building  provide  good  opportunities  to  establish 
team  familiarity  with  the  appraisal  plan.  This  includes  such  items  as  appraisal 
objectives,  organizational  scope,  reference  model  scope,  and  the  schedule, 
resources,  and  constraints  for  conducting  the  appraisal.  Team  member  input 
can  be  obtained  to  refine  or  complete  the  contents  of  the  appraisal  plan. 
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Implementation 

Guidance 

Analysis  of 

Objective 

Evidence 


Analysis  of  the  objective  evidence  provided  by  the  appraised  organization, 
such  as  questionnaire  responses  or  worksheets  summarizing  objective 
evidence,  can  be  accomplished  following  or  as  an  integrated  part  of  appraisal 
team  preparation  and  training. 

Team  members  should  become  familiar  with  the  instruments  (e.g., 
questionnaires,  PH  database)  to  be  used  as  data  collection  sources  during  the 
appraisal.  Demonstrations  or  exercises  using  the  data  collection  tools  and 
methods  planned  for  the  appraisal  should  be  used  to  provide  appraisal  team 
members  with  an  opportunity  to  practice  techniques  for  data  recording, 
verification,  and  analysis.  This  may  include  mechanisms  such  as  wall  charts, 
spreadsheets,  or  data  reduction  tools.  The  more  familiarity  and  comfort  that 
can  be  obtained  with  these  tools  in  advance,  the  greater  the  savings  in  team 
efficiency  during  the  appraisal  on-site  phases. 
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1.3.3  Prepare  Team  (continued) 


implementation 

Guidance 

Roles  and 
Responsibilities 


Ti  iraisal  team  leader  should  assign  and  explain  team  member  roles  and 
responsibilities  to  be  performed  during  the  appraisal.  Typical  roles  to  be 
assigned  include: 

Organizational  Unit  Coordinator.  The  Organizational  Unit  Coordinator 
handles  on-site  logistics  and  provides  technical,  administrative,  and  logistical 
support  to  the  appraisal  team  leader.  This  usually  includes  activities  such  as 
coordinating  schedules,  notifying  participants,  arranging  adequate  facilities 
and  resources,  obtaining  requested  documentation,  and  arranging  catering.  He 
or  she  may  also  coordinate  or  provide  clerical  support  to  the  team.  This  role  is 
often  assigned  to  one  or  more  members  of  the  organizational  unit.  The 
Organizational  Unit  Coordinator  may  be  one  of  the  appraisal  team  members, 
or  this  role  may  be  assigned  to  other  site  personnel. 


Librarian:  The  librarian  manages  the  inventory  of  appraisal  documents, 
coordinates  requests  for  additional  documentation  evidence,  and  returns 
documents  at  the  end  of  the  appraisal.  This  role  can  be  fulfilled  by  an 
appraisal  team  member  or  by  a  member  of  the  support  staff. 


Process  Area  Mini-Teams:  Mini-teams  take  the  lead  for  data  collection  in 
assigned  PAs.  They  ensure  that  information  collected  during  a  data  gathering 
session  covers  their  PAs,  request  additional  information  needed  relative  to 
their  PAs,  and  record  the  work  performed  by  individual  appraisal  team 
members  pertaining  to  their  PAs. 

Mini-teams  typically  consist  of  two  or  three  members.  Mini-team  assignments 
can  be  made  based  on  several  factors,  including 

•  related  PAs  (e.g.,  PA  categories) 

•  composition  mix  of  mini-team  members  (e.g.,  discipline  experience, 
appraisal  experience) 

Facilitator:  The  facilitator  conducts  interviews,  asking  questions  of  interview 
participants. 

Timekeeper:  The  timekeeper  is  responsible  for  tracking  time  and  schedule 
constraints  during  interviews  and  other  activities. 

Observer:  Due  to  the  confidentiality  required  during  an  appraisal  and  the 
cohesiveness  needed  to  participate  in  appraisal  activities,  observers  are  not 
permitted  to  participate  in  the  appraisal  processes.  The  only  exception  is  an 
observer  who  is  authorized  by  the  CMMI  Steward  to  observe  a  candidate  Lead 
Appraiser’s  performance  as  appraisal  team  leader  or  to  perform  an  audit  as 
part  of  the  quality  audit  function  of  the  Steward. 
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GSS  and  CPM  Implementation  Guidance 

When  using  the  MDD  for  Government  source  selection  and  contract  process  monitoring,  the 
following  should  be  considered: 

Guidance  for  Appraisal  team  members  must  receive  orientation  on  the  sponsor’s  supplier 
GSS  and  CPM  selection  or  contract  performance  management  goals  and  objectives  for  the 
SCAMPI  and  on  how  the  results  of  the  appraisal  are  intended  to  be  used  after 
the  appraisal  activities  are  concluded. 

Acquisition  regulations,  source  selection  requirements,  or  contract 
requirements  may  necessitate  observers  accompanying  the  appraisal  team  for 
a  variety  of  reasons  (e.g.,  as  representatives  of  the  contracting  office  to  ensure 
integrity  of  the  source  selection  activities;  as  alternate  members,  ready  to 
augment  the  core  appraisal  team  if  any  individual  member  becomes 
incapacitated  or  unavailable;  as  representatives  of  the  appraised  organization 
or  prime  contractor).  While  observers  may  be  required  during  appraisal 
activities,  they  can  have  no  role  as  an  active  participant  in  the  actual  appraisal 
events,  as  described  in  the  implementation  guidance  in  activity  1.3.3.  Any 
observers  who  are  required  to  accompany  the  appraisal  team  should  receive 
the  same  orientation  as  the  rest  of  the  team.  Observers  should  also  be  included 
in  any  model  and  appraisal-method  training  the  team  receives  to  gain  a  better 
understanding  of  what  they  are  observing. 
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1 .4  Obtain  and  Analyze  Initial  Objective  Evidence 


Purpose 

Entry  Criteria 

Inputs 

Activities 

Outputs 

Outcome 

Exit  Criteria 


Obtain  information  that  facilitates  site-specific  preparation  and  an 
understanding  of  the  implementation  of  model  practices  across  the 
organizational  unit.  Identify  potential  issues,  gaps,  or  risks  to  aid  in  refining 
the  plan.  Strengthen  understanding  of  the  organization’s  operations  and 
processes. 


•  Appraisal  input  received 

•  Sponsor  authorization  to  proceed 

•  Availability  of  practice  implementation  data  for  organizational  unit 


•  Practice  implementation  data  for  organizational  unit 

•  Identified  participants 


1.4.1  Prepare  Participants 

1.4.2  Administer  Instruments 

1.4.3  Obtain  Initial  Objective  Evidence 

1.4.4  Inventory  Objective  Evidence 


•  Completed  instruments 

•  Data  analyses  results  (data  summaries,  questionnaire  results,  etc.) 

•  Identification  of  additional  information  needed 

•  Prepared  participants 

•  Initial  set  of  objective  evidence 


•  Initial  objective  evidence  has  been  collected,  organized,  and  recorded. 

•  Potentially  important  areas  of  needed  information  have  been  noted. 

•  The  team  has  a  deeper  understanding  of  the  organizational  unit’s 
operations  and  processes. 

•  The  team  is  ready  to  make  detailed  plans  for  data  collection. 


•  All  objective  evidence  captured  during  this  activity  has  been  recorded  for 
later  use. 

•  High-priority  areas  for  additional  data  collection  have  been  identified. 

•  The  level  of  sufficiency  of  the  objective  evidence  to  support  the  appraisal 
is  determined. 


Continued  on  next  page 
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1.4  Obtain  and  Analyze  Initial  Objective  Evidence  (continued) 


Key  Points 


Tools  and 
Techniques 


Metrics 


Verification  and 
Validation 


Records 


Tailoring 


Gather  high-leverage  o1"  active  evidence.  The  amount  of  initial  objective 
evidence  provided  by  the  organization  will  determine  the  proportion  of 
evidence  that  must  be  discovered  (versus  ve  Tied)  during  the  appraisal. 
Maximizing  time  spent  in  verification,  versus  discovery,  is  a  key  performance 
objective  for  the  appraisal  process. 


Automated  support  for  questionnaires,  including  data  reduction  tools,  may 
be  available  to  make  the  data  analysis  activity  more  efficient. 

Breaking  into  mini-teams  to  review  data  related  to  specific  PAs  is  a  way 
to  ensure  completeness  in  the  data. 


The  number  of  practices  for  which  complete  objective  evidence  is 
available 

The  number  of  questionnaire  respondents  reported  in  the  Appraisal 
Record 

The  calendar  time  and  effort  expended  for  this  activity  compared  to  the 
planned  values 


Where  the  team  includes  members  of  the  appraised  organization,  these 
members  should  be  used  to  help  understand  the  initial  objective  evidence 
provided  to  prevent  misinterpretation  of  terms  or  special  conditions. 
Inconsistencies  and  contradictions  among  the  items  provided  in  initial 
objective  evidence  should  be  identified  and  recorded  for  resolution. 


Records  of  this  process  include  completed  and/or  summarized 
questionnaires,  profiles,  and  surveys. 

Lists  of  information  needed  should  be  maintained  and  used  as  an  input  to 
the  later  data  collection  activities. 

Calendar  time  and  effort  expended  in  this  activity  should  be  recorded  and 
compared  to  the  plan.  These  data  will  be  part  of  the  Appraisal  Record. 


A  variety  of  methods  can  be  used  to  collect  initial  data,  including 

•  a  site  information  package  prepared  by  representatives  of  the  organization 

•  a  presentation  on  the  process  improvement  program  and  its  accomplishments 

•  specialized  or  general  questionnaires  focused  on  practice  implementation 

The  use  of  additional  instruments  is  dependent  on  the  results  of  the  analysis  of 
available  data  and  the  results  of  process  1.5,  Prepare  for  Collection  of 
Objective  Evidence. 


Continued  on  next  page 
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1 .4  Obtain  and  Analyze  Initial  Objective  Evidence  (continued) 


Interfaces 
with  Other 
Processes 


Summary  of 
Activities 


This  process  plays  a  critical  role  in  the  planning  and  preparation  processes. 
The  information  generated  in  this  process  provides  the  most  important 
opportunity  to  reset  expectations  and  plans  with  the  appraisal  sponsor,  if 
initial  assumptions  about  the  availability  of  objective  evidence  turn  out  to  be 
in  error.  It  will  also  provide  the  basis  of  data  collection  planning. 


The  appraisal  team  leader  works  with  representatives  of  the  organization  to 
obtain  an  initial  data  set  that  represents  an  inventory  of  the  objective  evidence 
pertaining  to  the  implementation  of  each  instantiation  of  each  practice  within 
the  appraisal  scope.  This  initial  data  set  is  first  reviewed  by  the  appraisal  team 
leader  for  a  high-level  assessment  of  adequacy  and  completeness.  The 
appraisal  team  leader  or  appraisal  team  then  performs  a  more  detailed  analysis 
to  use  as  input  for  planning  the  data  collection  and  verification  activities  that 
will  occur  when  they  arrive  on  site.  Finally,  a  record  is  created  that  reflects  a 
detailed  accounting  of  any  missing  objective  evidence.  This  record  is  used  as 
primary  input  for  the  generation  of  the  data  collection  plan. 


♦ 
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1 .4  Obtain  and  Analyze  Initial  Objective  Evidence 


GSS  and  CPM  implementation  Guidance 

When  using  the  MDD  for  Government  source  selection  and  contract  process  monitoring,  the 
following  should  be  considered: 


Guidance  for 
GSS 


Depending  on  the  approach  to  request  and  receive  the  initial  objective 
evidence,  there  may  be  a  requirement  to  include  wording  (Instructions  in 
Section  L)  in  the  RFP.  It  may  be  appropriate  for  the  Section  L  to  request 
submittal  of  PH  data  to  facilitate  a  verification-based  evaluation  instead  of  a 
more  discovery-oriented  evaluation.  Various  acquisition  centers  have  differing 
approaches  to  requesting  this  data.  Determine  whether  your  source  selection 
policies  require  this  data  to  be  submitted  as  part  of  the  proposal  or  proposal- 
associated  data,  or  whether  the  appraisal  team  may  request  this  data  outside 
the  control  of  the  source  selection  process.  Some  Source  Selection  Offices 
will  allow  submittal  of  the  required  data  as  part  of  the  proposal,  but  not  under 
the  constraints  of  page  limits  associated  with  the  rest  of  the  proposal  content. 
The  RFP  information  could  also  be  affected  by  the  decision  to  make  the 
appraisal  team  part  of  the  SSEB  or  keep  it  separate.  If  separate,  then  obviously 
the  data  cannot  be  submitted  as  part  of  the  proposal.  In  that  case,  alternative 
delivery  strategies  would  have  to  be  defined  in  consonance  with  the  intended 
implementation  of  SCAMPI  results  into  the  overall  acquisition. 


Guidance  for  No  additional  guidance. 

CPM 


♦ 
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1 .4.1  Prepare  Participants 


Activity 

Description 


Required 

Practices 

Parameters 
and  Limits 


Members  of  the  organization  who  participate  in  the  appraisal  must  be 
informed  of  their  role,  and  the  expectations  the  sponsor  and  appraisal  team 
have.  This  is  typically  accomplished  through  a  briefing  where  the  appraisal 
team  leader  provides  an  overview  of  the  appraisal  process,  purpose,  and 
objectives.  Specific  information  about  the  scheduled  events  and  the  locations 
where  they  occur  is  also  communicated  during  this  presentation,  as  well  as 
through  ongoing  contact  between  the  Organizational  Unit  Coordinator  and  the 
members  of  the  organization. 


•  Brief  appraisal  participants  on  the  appraisal  process. 

•  Provide  orientation  to  appraisal  participants  on  their  roles  in  the  appraisal. 


The  orientation  provided  to  appraisal  participants  must  occur  some  time  prior 
to  their  participation  to  allow  participants  to  confirm  their  availability  and  to 
prepare  for  their  participation. 

The  preparation  of  appraisal  participants  may  be  accomplished  via 
video/teleconference  if  desired. 


Continued  on  next  page 
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1.4.1  Prepare  Participants  (continued) 


Optional 

Practices 


Implementation 

Guidance 


Page  11-100 


Provide  orientation  on  the  documentation  of  PIIs  and  any  specific  instruments 
used,  so  the  appropriate  people  in  the  organization  can  document  the  initial 
objective  evidence  to  be  used  in  the  appraisal. 


Depending  on  the  appraisal  usage  mode  (e.g.,  supplier  selection  versus 
internal  process  improvement),  various  types  of  communications  may  be  used. 
In  the  internal  process  improvement  usage  mode,  the  importance  of 
management  sponsorship  within  the  organization  will  likely  lead  the  appraisal 
team  leader  to  work  with  senior  management  to  help  demonstrate  commitment 
to  the  appraisal  process  as  well  as  the  process  improvement  work  that  will 
follow.  In  the  supplier  selection  usage  mode,  the  possibility  of  the  same  team 
visiting  multiple  organizations  adds  coordination  tasks  and  communication 
channels  as  well. 

Preparation  of  appraisal  participants  should  also  include  informing  them  of  the 
need  to  provide  accurate  and  complete  information  on  instruments.  In  addition 
to  assisting  with  appraisal  accuracy,  this  can  help  to  ensure  sufficient  coverage 
of  reference  model  practices  and  reduce  the  amount  of  time  necessary  for 
follow-up  interviews.  The  investment  in  initial  population  of  complete 
instruments,  such  as  PIIs,  questionnaires,  or  mapping  tables,  can  be  recovered 
by  reduced  effort  in  the  reuse  of  assets  for  subsequent  appraisals. 


♦ 
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1 .4.1  Prepare  Participants 


GSS  and  CPM  Implementation  Guidance 

When  using  the  MDD  for  Government  source  selection  and  contract  process  monitoring,  the 
following  should  be  considered: 

Guidance  for  For  GSS  and  CPM  the  process  may  differ  slightly  from  other  SCAMPI 
GSS  and  CPM  appraisals,  and  the  participants  should  be  briefed  on  their  role  in  the  appraisal. 

Address  any  issues  peculiar  to  GSS,  such  as  constraints  on  feedback 
(preliminary  findings  and  final  findings,  ratings,  and  the  time  frame  of  the 
feedback)  imposed  or  updating  the  formal  proposal  based  on  clarifications 
provided  during  the  appraisal.  For  CPM,  address  any  sensitive  issues  related 
to  contract  incentives  and  how  they  might  influence  the  verification  or 
discovery  process. 
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1.4.2  Administer  Instruments 


Activity 

Description 


Required 

Practices 

Parameters 
and  Limits 


This  activity  involves  the  administration  of  instruments  for  the  appraisal  that 
are  additional  to  the  input  data  (such  as  process  implementation  indicators 
provided  by  the  organization  as  input  to  the  appraisal).  It  includes  the  use  of 
structured  techniques  and  work  aids  (e.g.,  surveys,  questionnaires,  or  an 
objective  evidence  database)  to  assist  the  organizational  unit  in  characterizing 
their  process  and  supporting  objective  evidence  in  terms  of  model  practices. 

A  practice-based  questionnaire  is  also  a  commonly  used  instrument  during 
appraisals.  Such  questionnaires  typically  have  a  series  of  focused  questions, 
each  one  providing  an  opportunity  for  the  respondent  to  answer  a  closed- 
ended  question  about  a  practice.  In  addition,  the  respondent  is  given  an 
opportunity  to  write  a  clarifying  comment  that  serves  to  elaborate  on  the 
closed-ended  response. 


Administer  appraisal  instruments  for  the  entry  of  data  by  appraisal 
participants. 


The  application  of  this  activity  to  generate  instrument  data  to  support  the  data 
collection  plan  is  limited  to  the  instruments  identified  in  the  data  collection 
plan.  Instruments  are  typically  administered  by  representatives  of  the  appraisal 
team.  The  appraisal  team  leader  is  responsible  for  negotiating  additional  time 
and  resources  if  the  data  provided  using  instruments  is  incomplete.  It  is  also 
the  responsibility  of  the  appraisal  team  leader  to  avoid  requesting  duplicate 
data  entry  on  multiple  instruments.  No  organization  should  be  asked  to 
provide  the  same  information  in  two  (or  more)  formats. 

Whatever  vehicle  is  used,  the  resultant  data  must  provide  information  about 
the  extent  of  the  implementation  of  model  practices  in  the  organizational  unit 
and  the  sampled  projects. 


Continued  on  next  page 
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1.4.2  Administer  Instruments  (continued) 


Optional 

Practices 


Implementation 

Guidance 


Page  11-104 


Establish  an  organizational  asset  (or  rely  on  an  existing  one)  that  documents 
and  maintains  the  traceability  of  implemented  practices  to  model  practices. 

Conduct  a  workshop  to  document  the  PIIs  for  the  organization. 


The  use  of  instruments  to  gather  written  information  from  members  of  the 
organization  provides  a  relatively  low-cost  data  collection  technique  when 
done  well.  Data  of  this  type  tend  to  be  most  useful  when  provided  early  in  the 
appraisal  conduct,  and  can  lead  to  valuable  insights  about  where  data  may  be 
sought  during  subsequent  data  collection  events.  Since  there  is  limited 
opportunity  for  elaboration  and  “branching”  to  related  topics,  responses  to 
instruments  can  sometimes  raise  more  questions  than  they  answer  for  the 
appraisal  team  member  trying  to  interpret  the  responses.  Furthermore, 
instruments  that  contain  excessive  jargon  or  complicated  terminology  may 
hinder  data  collection  rather  than  help.  Confused  respondents  will  do  then- 
best  to  answer  the  question  they  don’t  quite  understand,  and  the  response  is 
interpreted  based  on  the  question  that  was  intended.  Having  a  knowledgeable 
person  present  during  the  administration  of  an  instrument  can  help  mitigate 
the  risk  of  miscommunication. 

One  of  the  attractive  features  of  instruments  for  the  purpose  of  data  collection 
is  that  they  can  be  used  to  establish  a  “scoring  scheme”  that  reduces  the 
burden  of  interpretation  for  the  recipient  of  the  data.  Such  schemes  do  not 
exist  for  SCAMPI,  and  the  use  of  a  shortcut  of  this  type  is  a  violation  of  the 
principle  that  focuses  rating  judgments  on  the  goals  of  the  PAs  in  CMMI 
models.  The  practices  found  in  CMMI  models  are  Expected  Components, 
while  the  goals  in  the  models  are  Required  Components.  While  the 
satisfaction  of  a  PA  goal  is  predicated  on  the  implementation  of  practices 
found  in  the  model  (or  acceptable  alternatives),  there  is  no  strict  aggregation 
scheme  that  allows  one  to  infer  goal  satisfaction  based  on  practice 
implementation  alone.  Rating  judgments  are  based  on  multiple  sources  of 
objective  evidence  and  the  reasoned  consideration  of  strengths  and 
weaknesses,  in  aggregate. 

Whenever  possible,  documents  mentioned  in  the  responses  to  questionnaires 
or  other  instruments  should  be  requested  for  team  review  early  in  the  process, 
so  that  any  misleading  references  will  not  cause  undue  confusion  later. 


♦ 
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1.4.2  Administer  Instruments 


GSS  and  CPM  implementation  Guidance 

When  using  the  MDD  for  Government  source  selection  and  contract  process  monitoring,  the 
following  should  be  considered: 

Guidance  for  Depending  on  your  source  selection  environment,  all  data  available  to  the 
GSS  team  prior  to  the  on-site  portion  of  the  appraisal  may  need  to  be  included  in 

the  RFP,  including  requests  for  instruments  (maturity  questionnaires,  PIIs, 
etc.). 

The  data  collection  plan  may  be  appropriate  for  inclusion  into  the  Source 
Selection  Plan  as  a  means  for  planning  and  documenting  the  approach  to 
obtain  the  initial  instrument  data. 

It  may  be  appropriate  to  require  documents  referenced  in  instrument  responses 
to  be  submitted  with  the  instrument  response.  This  further  aids  the  appraisal  of 
the  process  for  completeness  and  can  reduce  the  on-site  appraisal  workload.  If 
appropriate  to  your  source  selection  environment,  include  this  requirement  in 
the  RFP  Section  L. 

Guidance  for  No  additional  guidance. 

CPM 

♦ 
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1.4.3  Obtain  Initial  Objective  Evidence 


Activity 

Description 


Required 

Practices 

Parameters 
and  Limits 


Optional 

Practices 


The  appraisal  team  leader  will  request  that  the  organization  provides  detailed 
data  on  the  implementation  of  practices  in  the  organization.  The  appraisal 
team  leader  is  free  to  specify  the  format  to  be  used  and  the  level  of  detail  to  be 
provided,  knowing  that  anything  that  is  not  provided  in  advance  must  be 
collected  later  in  the  appraisal  process.  There  are  no  minimum  requirements 
set  by  the  method  with  respect  to  completeness  or  detail  in  this  initial  data  set. 
However,  the  effort  required  to  conduct  a  SCAMPI  appraisal  is  a  direct 
function  of  the  amount  of  data  available  to  the  team  at  the  beginning  of  the 
process.  Before  the  appraisal  outputs  can  be  created,  the  team  will  need  to 
verify  objective  evidence  for  each  instantiation  of  each  practice  within  the 
scope  of  the  appraisal.  For  detailed  requirements  on  the  sufficiency  of  data, 
refer  to  process  2.2,  Verify  and  Validate  Objective  Evidence. 

The  use  of  a  completely  populated  PH  database  is  desirable  but  not  essential  at 
this  stage  in  the  appraisal  process.  The  appraisal  team  leader  must  provide  an 
opportunity  for  the  organization  to  provide  it,  but  will  not  require  it  unless  the 
sponsor  has  agreed  that  this  will  be  a  verification-oriented  appraisal  (as 
opposed  to  a  discovery-oriented  appraisal). 

A  “mapping”  of  implemented  practices  and  model  practices  is  required,  and 
may  be  generated  using  questionnaires  (see  activity  1.4.2). 


Obtain  documentation  reflecting  the  implementation  of  model  practices  within 
the  organizational  unit  and  sampled  projects. 


At  a  minimum,  the  organization  must  provide  a  list  of  documents  that  are 
relevant  to  understanding  the  processes  in  use  in  the  organizational  unit  and 
the  sampled  projects.  This  list  must  be  mapped  to  the  model  practices  that  are 
in  the  scope  of  the  appraisal. 


A  list  of  terms  and  important  jargon  used  in  the  organizational  unit  may  be 
provided  to  the  team,  to  aid  in  communicating  with  the  members  of  the 
organization. 

A  complete  objective  evidence  database,  which  documents  the 
implementation  of  every  model  practice  (within  the  scope  of  the  appraisal)  in 
the  organizational  unit  and  the  sampled  projects,  may  be  provided  to  the  team 
in  advance. 

The  use  of  database  tools  specifically  built  to  support  a  process  appraisal  is 
highly  recommended. 


Continued  on  next  page 
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1.4.3  Obtain  initial  Objective  Evidence  (continued) 


Implementation 

Guidance 


Whether  collected  through  instruments,  the  review  of  documents,  attending 
presentations,  or  interviews,  the  data  used  for  an  appraisal  is  related  to  the 
practices  of  the  reference  model.  For  every  practice  within  the  model  scope  of 
the  appraisal,  and  for  every  instance  of  each  practice,  objective  evidence  is 
used  as  the  basis  for  appraisal  team  determinations  of  the  extent  to  which  the 
practice  is  implemented.  Indicators  that  substantiate  practice  implementation 
include 

•  direct  artifacts,  which  represent  the  primary  tangible  output  of  a  practice. 
These  are  typically  listed  in  CMMI  models  as  typical  work  products.  One 
or  more  direct  artifacts  may  be  necessary  to  verify  the  implementation  of 
associated  model  practices. 

•  indirect  artifacts,  which  represent  artifacts  that  are  a  consequence  of 
performing  the  practice,  but  not  necessarily  the  puipose  for  which  it  is 
performed.  These  are  typically  things  like  meeting  minutes,  review 
results,  or  written  commuhications  of  status. 

•  affirmations,  which  are  oral  or  written  statements  confirming  the 
implementation  of  the  practice.  These  are  typically  validated  using 
interviews,  questionnaires,  or  other  means. 

Prior  to  the  data  collection  activities  carried  out  by  the  appraisal  team,  an 
initial  data  set  is  usually  created  by  the  appraised  organization.  This  data  set 
contains  descriptions  of  the  objective  evidence  available  for  the  team  to 
examine,  complete  with  references  to  documentation  and  identification  of  the 
personnel  who  can  provide  relevant  affirmations.  This  instrument  provides  the 
baseline  of  objective  evidence  for  the  appraisal.  Most  organizations 
experienced  in  process  improvement  will  already  have  this  type  of  data  on 
hand,  as  they  will  have  used  it  to  track  their  progress. 

Artifacts  may  be  obtained  as  hard  copies,  soft  copies,  or  hyperlinks  to  where 
these  documents  reside  in  a  Web-based  environment.  If  hyperlinks  are  used, 
the  accessibility  of  artifacts  via  these  links  should  be  verified  in  the  appraisal 
environment.  For  example,  appraisal  team  access  could  be  inhibited  by  invalid 
references  or  firewalls. 

The  initial  data  set  forms  the  basis  for  planning  the  data  collection  activities, 
including  interviews  and  presentations  on  site.  Any  objective  evidence  that  is 
not  identified  in  advance  of  the  team’s  arrival  will  need  to  be  sought  by  the 
team  members  once  they  arrive  on  site.  This  process  of  “discovering” 
whether,  and  how,  the  organization  has  addressed  a  given  practice  in  the 
model  can  be  quite  time  consuming,  and  it  is  often  difficult  to  predict  how 
long  it  will  take. 


♦ 
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1.4.3  Obtain  Initial  Objective  Evidence 


GSS  and  CPM  Implementation  Guidance 

When  using  the  MDD  for  Government  source  selection  and  contract  process  monitoring,  the 
following  should  be  considered: 

Guidance  for  Once  again,  the  inclusion  of  the  requirement  for  submittal  of  practice 
GSS  implementation  data  (PII  databases,  instruments,  surveys)  should  be  included 

in  the  RFP  if  appropriate  to  your  source  selection  environment.  If  not,  this 
data  is  requested  by  the  appraisal  team  outside  the  control  of  the  source 
selection. 

Many  source  selections  require  X  copies  of  proposals.  Determine  the  number 
of  copies  required  for  the  appraisal  data  and,  if  different  from  the  proposal 
requirements,  be  specific  in  the  request  for  the  data  (e.g.,  25  copies  of  general 
proposal  data,  2  copies  of  data  for  CMMI  appraisal). 

Be  sure  to  verify  the  appropriateness  of  the  request  for  data,  especially  in 
regards  to  official  discussions  in  the  context  of  your  source  selection 
environment. 


Guidance  for  No  additional  guidance. 

CPM 


♦ 
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1 .4.4  Inventory  Objective  Evidence 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Optional 

Practices 


The  analysis  of  the  initial  data  set  provides  critical  new  information  for  the 
overall  planning  of  the  appraisal  and  forms  the  basis  for  the  detailed  data 
collection  plan  that  must  be  developed  before  the  on-site  data  collection 
begins.  The  analysis  of  initial  objective  evidence  at  this  stage  is  focused 
primarily  on  the  adequacy  and  completeness  of  information  and  the 
implications  for  future  data  collection.  The  results  of  this  analysis  will  be  the 
primary  basis  for  determining  the  extent  to  which  the  appraisal  will  be  one  of 
verification  or  discovery. 


•  Examine  the  initial  set  of  objective  evidence  provided  by  the 
organizational  unit. 

•  Determine  the  extent  to  which  additional  information  is  needed  for 
adequate  coverage  of  model  practices. 


Information  provided  by  the  organizational  unit  must  be  detailed  enough  to 
understand  the  extent  to  which  each  type  of  objective  evidence  (direct 
artifacts,  indirect  artifacts,  and  affirmations)  is  available  for  each  process 
instantiation,  for  each  model  practice  within  the  scope  of  the  appraisal.  This 
initial  review  of  objective  evidence  identifies  model  practices  for  which  the 
team  has 

•  strong  objective  evidence 

•  no  objective  evidence 

•  conflicting  objective  evidence 

•  anomalous  objective  evidence 

•  insufficient  objective  evidence 

Key  documents  are  identified  that  can  be  used  to  gain  insight  regarding  a 
number  of  model  practices.  These  are  potential  high-leverage  documents  that 
may  be  good  candidates  for  pre-on-site  review  by  team  members. 


Review  the  initial  objective  evidence  with  members  of  the  engineering 
process  group. 


Continued  on  next  page 
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1.4.4  Inventory  Objective  Evidence  (continued) 


Implementation 

Guidance 


Members  of  the  team  may  choose  to  summarize  the  extent  of  practice 
implementation  at  the  discretion  of  the  appraisal  team  leader.  However,  the 
objective  of  this  activity  is  to  determine  how  much  additional  data  team 
members  will  need  to  complete  their  work.  It  is  recommended  that  the 
appraisal  team  leader  establish  an  expectation  with  the  sponsor  that  the  results 
of  this  analysis  will  form  the  basis  for  a  revised  schedule  estimate.  If  the 
initial  objective  evidence  is  lacking  in  completeness  and  detail,  the  team  will 
be  forced  to  seek  more  information  during  the  on-site  data,  collection,  unless 
corrective  actions  are  taken  before  that  time. 

It  is  important  to  keep  all  stakeholders  focused  on  the  fact  that  SCAMPI  is 
intended  as  a  benchmarking  appraisal.  This  method  is  not  well  suited  for 
organizations  that  have  very  limited  understanding  of  CMMI.  Such 
organizations  may  not  yet  have  a  clear  idea  of  how  the  practices  described  in 
CMMI  models  ought  to  be  implemented  to  meet  their  specific  business  needs. 
Deciding  on  a  reasonable  implementation  of  the  practices,  and  working  to 
ensure  that  they  are  enacted  on  projects  throughout  the  organization,  are 
activities  that  precede  a  benchmarking  appraisal.  A  different  type  of  appraisal 
(Class  B  or  C)  is  probably  going  to  be  more  valuable  if  the  objective  of  the 
sponsor  is  to  begin  the  process  of  understanding  what  CMMI  could  mean  for 
the  organization.  It  is  not  reasonable  to  schedule  a  two-week  appraisal  and 
expect  to  collect  all  of  the  data  required  for  benchmarking  during  the  on-site 
data  collection. 

The  appraisal  team  leader  often  reviews  the  initial  data  set  provided  by  the 
organization  prior  to  assembling  the  team  for  its  first  meeting,  to  identify 
areas  where  additional  data  will  be  needed  and  to  assess  the  feasibility  of  the 
planned  appraisal  schedule.  This  readiness  review  should  be  conducted  prior 
to  finalizing  the  appraisal  schedule,  and  may  comprise  a  “Go/No  Go”  decision 
for  the  appraisal  in  some  situations.  The  appraisal  team  will  then  review  the 
initial  objective  evidence  in  more  detail  (typically  toward  the  end  of  the  team¬ 
training  event)  to  begin  formulating  plans  for  how  missing  evidence  will  be 
collected,  and  for  the  verification  of  the  entire  data  set.  This  preliminary 
readiness  review  is  the  basis  for  the  data  collection  plan,  which  is  described  in 
the  next  process.  Prepare  for  Collection  of  Objective  Evidence. 


Continued  on  next  page 
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1 .4.4  Inventory  Objective  Evidence  (continued) 


Implementation 

Guidance 

(continued) 


The  appraisal  team  leader  generates  a  list  of  additional  information  needed. 
The  results  of  the  analysis  of  initial  objective  evidence  are  documented  as  an 
input  to  the  data  collection  plan.  The  use  of  an  integrated  appraisal  tool  to 
annotate  the  set  of  initial  objective  evidence  will  permit  the  automated 
tracking  of  information  needs,  and  will  aid  in  the  compilation  of  a  detailed 
data  collection  plan.  Where  the  completeness  of  initial  objective  evidence  is 
insufficient  to  conduct  the  appraisal  under  the  original  schedule,  the  results  of 
this  activity  form  an  important  basis  for  renegotiating  the  appraisal  schedule 
in  some  cases. 

The  adequacy  of  objective  evidence  relative  to  model  practices  is  typically 
determined  using  a  software  tool  of  some  sort,  either  one  built  for  use  on 
appraisals,  or  a  spreadsheet  template.  However,  paper  forms  and  wall  charts 
may  be  used  if  preferred. 


♦ 
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1 .4.4  Inventory  Objective  Evidence 


GSS  and  CPM  implementation  Guidance 

When  using  the  MDD  for  Government  source  selection  and  contract  process  monitoring,  the 
following  should  be  considered: 

Depending  on  the  clarity  and  completeness  of  the  initial  data  submitted,  you 
may  rely  on  the  on-site  data  collection  activity  to  fill  the  void  or  you  may 
decide  to  issue  an  Evaluation  Notice  through  the  source  selection  process  to 
request  more  complete  data  prior  to  the  on-site  data  collection.  This  again 
would  reduce  the  on-site  appraisal  workload. 

The  inventory  of  objective  evidence  may  also  be  used  to  support  the  “Quick 
Look”  process,  if  it  is  employed  in  your  source  selection  environment.  The 
Quick  Look  process  is  a  high-level  check  to  determine  overall  proposal 
responsiveness  to  the  RFP  requirements. 

There  may  be  source-selection  constraints  on  the  level  of  interaction  allowed 
with  the  offeror  during  this  time,  so  some  of  the  benefits  derived  from 
working  with  the  appraised  organization  to  ensure  completeness  of  the 
objective  evidence  may  not  be  realizable. 

In  source  selections,  the  initial  data  may  not  be  available  prior  to  team 
formation  and  hence  may  not  be  available  for  team  training. 

There  may  be  little  flexibility  in  the  schedules  for  the  on-site  data  collection 
due  to  schedule  constraints  of  the  source  selection.  The  team  will  need  to 
work  any  schedule  changes  caused  by  major  submittal  deficiencies  through 
the  source  selection  authorities  to  determine  the  appropriate  course  of  action 
(disqualification,  schedule  change,  etc.). 

Once  the  appraisal  team  leader  determines  additional  information  needed 
based  on  the  review  of  the  submitted  initial  information,  he/she  will  need  to 
determine  the  proper  method  for  notifying  the  offeror  of  the  requested 
additional  data  consistent  with  source  selection  processes. 


Guidance  for 
GSS 


Guidance  for  No  additional  guidance. 

CPM 


♦ 
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1 .5  Prepare  for  Collection  of  Objective  Evidence 


Purpose  Plan  and  document  specific  data  collection  strategies,  including 


•  sources  of  data 

•  tools  and  techniques  to  be  used 

•  contingencies  to  manage  risk  of  insufficient  data 

Entry  Criteria 

•  Sponsor  commitment  to  proceed  with  the  appraisal  has  been  documented. 

•  Appraisal  objectives  and  constraints  have  been  documented. 

•  Initial  data  have  been  received  and  analysis  has  been  completed. 

Inputs 

•  Appraisal  plan 

•  PDs  for  the  organizational  unit 

•  Initial  objective  evidence  review 

•  Data  collection  status 

Activities 

1.5.1  Perform  Readiness  Review 

1.5.2  Prepare  Data  Collection  Plan 

1.5.3  Replan  Data  Collection 

Outputs 

•  Confirmation  that  objective  evidence  collected  is  sufficient  to  proceed 

•  Initial  data  collection  plan 

•  Updates  to  the  plan  as  required 

Outcome  Finalized  data  collection  plan.  Team  members  are  aware  of  data  needs  and  the 

status  of  initial  data  available  to  them. 


Exit  Criteria  All  preparations  for  data  collection  by  the  team  have  been  made  and  the  data 
collection  plan  has  been  documented. 

Continued  on  next  page 
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1.5  Prepare  for  Collection  of  Objective  Evidence  (continued) 


Key  Points 


Tools  and 
Techniques 


Metrics 


Verification  and 
Validation 


Records 


Tailoring 


The  data  collected  is  the  most  important  input  the  team  receives.  Careful 
planning,  disciplined  tracking  against  the  plan,  and  effective  corrective  actions 
are  cornerstones  to  success  in  this  process. 


The  use  of  a  spreadsheet  to  record  and  track  the  data  collection  plan  is  a 
common  technique.  A  matrix  showing  the  practices  of  the  model,  or  questions 
to  be  asked,  arrayed  on  the  vertical  axis  and  the  sources  of  information 
arrayed  on  the  horizontal  axis  provides  a  simple  planning  and  tracking  tool.  A 
number  of  vendor-provided  data  management  tools  are  available  as  well. 


•  Estimated  and  tracked  calendar  time  and  effort  for  this  activity 

•  Planned  and  actual  number  of  data  sources  per  practice 

•  Planned  and  tracked  number  of  scripted  questions  per  interview 

•  Planned  and  tracked  number  of  scripted  questions  per  PA 

•  Percentage  of  planned  coverage  achieved,  per  data  collection  event  or  PA 


The  data  collection  plan  should  be  summarized  and  reviewed  with  the  team  to 
ensure  that  appraisal  requirements  will  be  successfully  implemented  if  the 
plan  is  carried  forward.  Experienced  Lead  Appraisers  will  use  historical  data 
to  assess  the  feasibility  of  (and  risks  associated  with)  the  data  collection  plan. 


Planned  and  actual  coverage  of  practices  and  PAs  across  the  set  of  data 
collection  activities  should  be  recorded.  These  data  support  future  estimates 
and  corrective  actions  during  the  data  collection  activities. 


Replanning  is  performed  only  when  the  status  of  the  appraisal  conduct 
indicates  the  need  to  do  so. 

Additional  planning  and  coordination  steps  may  be  necessary  in  usage  modes 
where  data  collection  activities  will  occur  at  geographically  distributed  sites  or 
across  organizational  units  from  different  corporations  (such  as  in  a  Supplier 
Selection  usage  mode). 

SCAMPI  allows  great  flexibility  in  formulating  strategies  to  accomplish  the 
necessary  data  collection.  The  relative  emphasis  of  different  data  sources,  as 
well  as  data  types,  can  be  tuned  to  support  appraisal  objectives  relating  to  buy- 
in  as  well  as  coverage  and  rigor  for  important  areas. 


Continued  on  next  page 
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1 .5  Prepare  for  Collection  of  Objective  Evidence  (continued) 


Interfaces 
with  Other 
Processes 


Summary  of 
Activities 


The  data  collection  plan  is  an  essential  element  of  the  appraisal  plan.  The 
activities  described  here  rely  on  the  results  of  analyzing  the  initial  objective 
evidence  to  derive  a  plan  and  set  of  strategies  for  accomplishing  the  data 
collection  needed  to  meet  the  objectives  of  the  appraisal.  The  data  collection 
plan  developed  through  these  activities  is  reviewed  and  revised  on  a  continual 
basis  throughout  the  appraisal.  Dynamically  managing  the  inventory  of  data 
on  hand,  the  list  of  data  needed,  and  the  available  data  collection  opportunities 
are  processes  critical  to  the  success  of  the  appraisal. 


The  activities  in  this  process  serve  to  (a)  establish  the  initial  planning  baseline 
for  the  acquisition  of  objective  evidence  and  (b)  update  the  plan  to  account  for 
information  acquired  and  unexpected  developments.  Since  SCAMPI  is  a  data- 
intensive  method,  the  conduct  of  these  activities  in  accordance  with  the 
descriptions  provided  is  essential  to  the  successful  use  of  the  appraisal 
method. 


♦ 
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1 .5  Prepare  for  Collection  of  Objective  Evidence 
GSS  and  CPM  Implementation  Guidance 

When  using  the  MDD  for  Government  source  selection  and  contract  process  monitoring,  the 
following  should  be  considered: 

Guidance  for  Note  that  in  a  GSS  context,  contractor  offeror  teams  will  offer  different 
GSS  and  CPM  experience  and  expertise,  such  as  expertise  in  radars,  ship  building,  systems 
integration,  or  software  development.  This  can  influence  the  emphasis  and/or 
criticality  of  data  collected  for  the  various  reference  model  areas.  Accounting 
for  this  while  treating  all  contractor  teams  fairly  requires  a  data  collection 
strategy  that  takes  these  factors  into  account,  as  well  as  the  expected 
experience  and  expertise  required  for  the  potential  successful  offeror. 

Guidance  for  In  a  CPM  context,  the  data-collection  strategy  will  focus  on  areas  of  known  or 
CPM  perceived  risk  to  the  development  program. 

♦ 
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1 .5.1  Perform  Readiness  Review 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Optional 

Practices 


The  available  objective  evidence  is  reviewed  to  determine  the  extent  to  which 
the  requested  objective  evidence  has  been  gathered  and  whether  the  evidence 
is  sufficient  to  proceed  or  replanning  is  required. 


•  Determine  whether  the  objective  evidence  for  each  process  instance  is 
adequate  to  proceed  with  the  appraisal  as  planned. 

•  Review  the  feasibility  of  the  appraisal  plan  in  light  of  the  inventory  of 
objective  evidence  available. 


At  least  one  readiness  review  must  be  conducted  prior  to  assembling  the  team 
on  site  for  data  collection. 

Objective  evidence  for  all  PAs  within  the  scope  of  the  appraisal  must  be 
reviewed. 

Objective  evidence  for  all  projects  sampled  to  represent  the  organizational 
unit  must  be  reviewed. 


Integrating  a  readiness  review  with  the  team  training  event  will  allow  the 
appraisal  team  leader  to  involve  the  team  in  gaining  an  understanding  of  the 
data  available  to  support  the  appraisal. 


Continued  on  next  page 
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1.5.1  Perform  Readiness  Review  (continued) 


Implementation 

Guidance 


A  summary  of  the  inventory  of  objective  evidence  and  readiness  to  proceed 
should  be  reviewed  with  the  sponsor  or  his/her  designee.  If  insufficient 
objective  evidence  is  available,  the  appraisal  team  leader  may  need  to  initiate 
replanning  in  light  of  newly  discovered  constraints  (i.e.,  insufficient  data  to 
support  the  appraisal  as  planned).  Refer  to  activity  1.1.2,  Determine  Appraisal 
Constraints.  The  criteria  for  adequacy  will  depend  on  where  the  readiness 
review  occurs  in  the  schedule,  and  the  degree  of  verification  versus  discovery 
that  is  being  sought  for  the  on-site  phases  of  the  appraisal. 

More  than  one  readiness  review  is  likely  to  be  needed.  The  first  one  should  be 
performed  early  in  the  planning  phase,  and  the  second  should  be  performed 
once  the  objective  evidence  has  been  gathered  and  the  appraisal  is  ready  to 
start.  This  review  may  be  conducted  in  conjunction  with  the  team-training 
event. 

Thresholds  for  the  sufficiency  of  data  should  be  established  as  targets  to  be 
met  at  the  readiness  review.  For  example,  an  80%  threshold  may  be  used  to 
initiate  replanning  at  the  final  readiness  review.  That  is,  the  appraisal  team 
leader  establishes  an  expectation  with  the  sponsor  that,  if  more  than  20%  of 
the  objective  evidence  is  missing  at  the  time  of  team  training,  the  appraisal 
will  need  to  be  replanned.  However,  the  primary  objective  is  reducing  the  risk 
that  there  will  be  insufficient  objective  evidence  to  make  the  determinations 
required  by  the  appraisal  plan  in  the  time  allotted. 

The  readiness  review  is  a  key  event  whose  impact  should  not  be 
underestimated.  Failure  to  adequately  review  the  objective  evidence  available 
and  determine  the  impact  on  the  appraisal  plan  can  have  grave  consequences 
for  the  appraisal  team  during  the  on-site  period.  This  may  include  long  hours, 
exhaustion,  extensive  ad  hoc  data  collection  (discovery),  or  the  inability  to 
achieve  appraisal  objectives  within  defined  estimates  and  constraints. 


♦ 
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1 .5.1  Perform  Readiness  Review 


GSS  and  CPM  Implementation  Guidance 

When  using  the  MDD  for  Government  source  selection  and  contract  process  monitoring,  the 
following  should  be  considered: 

As  mentioned  in  the  guidance  for  GSS  in  activity  1.4.4,  Inventory  Objective 
Evidence,  the  initial  data  submittal  may  not  be  available  for  team  training. 
Source  selection  schedules  are  usually  compressed  to  the  extent  .that  the 
objective  should  be  having  a  trained  team  at  the  start  of  source  selection  in 
order  to  make  maximum  use  of  the  available  time  once  source  selection  starts. 

The  readiness  review  can  support  either  the  Quick  Look  or  the  determination 
of  the  overall  responsiveness  of  the  individual  offerors  (competitive  range),  if 
either  of  those  two  processes  is  used  in  your  source  selection. 

The  readiness  review  will  drive  the  data  collection  plan  for  the  appraisal. 
Based  on  the  outcome  of  the  review  (for  example,  “lack  of  sufficient  objective 
evidence”),  the  source  selection  authority  may  need  to  consider  one  of  the 
following  alternatives: 

•  disqualification 

•  renegotiating  the  appraisal  plan  (schedule  slips,  reduced  scope,  etc.) 

•  continuing  in  discovery  mode,  with  the  associated  potential  for 
lengthening  or  shortening  the  on-site  schedule  and/or  not  meeting 
appraisal  requirements  for  an  ARC  Class  A  SCAMPI 

The  team  leader  should  communicate  the  drawbacks  of  performing  a 
discovery-mode  appraisal  (more  error  prone,  time  consuming,  etc.)  to  the 
sponsor.  Evaluation  Notices,  or  the  equivalent  GSS  method  to  ask  for 
clarifications  or  additional  data,  may  also  be  issued  to  obtain  additional 
objective  evidence  in  order  to  reduce  the  amount  of  discovery  performed  on 
site. 

Guidance  for  No  additional  guidance. 

CPM 

♦ 


Guidance  for 
GSS 
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1.5.2  Prepare  Data  Collection  Plan 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


The  data  collection  activities  are  tailored  to  meet  the  needs  for  objective 
evidence  so  that  the  extent  of  practice  implementation  can  be  determined. 

For  practices  that  have  objective  evidence,  a  strategy  for  verifying  that 
evidence  will  be  formulated. 

For  practices  that  lack  objective  evidence,  a  strategy  for  discovering  that 
evidence  will  be  formulated. 

The  data  collection  plan  is  typically  embodied  in  a  number  of  different 
artifacts  used  during  the  appraisal  process.  The  appraisal  plan  includes 
information  about  the  site,  projects,  and  participants  involved  in  the  appraisal. 
This  is  the  highest  level  of  information  that  helps  document  and  communicate 
the  data  collection  plan.  Detailed  information  on  data  collection  can  be 
recorded  in  work  aids  that  manage  appraisal  data  and  in  the  appraisal 
schedule.  A  record  of  “information  needed”  items  is  the  most  detailed 
representation,  while  document  lists,  interview  schedules,  and  the  assignment 
of  PA  mini-teams  help  shape  the  strategy  for  obtaining  the  needed  data. 


•  Determine  participants  for  interviews. 

•  Determine  artifacts  to  be  reviewed. 

•  Determine  presentations/demonstrations  to  be  provided. 

•  Determine  team  roles  and  responsibilities  for  data  collection  activities. 

•  Document  the  data  collection  plan. 


For  every  instantiation  of  every  model  practice,  the  data  collection  plan  must 
specify  how,  when,  and  by  whom  the  objective  evidence  will  be  verified. 

For  instantiations  of  model  practices  that  have  not  been  addressed  in  the  initial 
objective  evidence,  the  data  collection  plan  must  specify  how  the  team  intends 
to  discover  the  presence  or  absence  of  objective  evidence  that  characterizes 
the  extent  of  implementation  for  that  practice. 


Continued  on  next  page 


CMU/SEI-2002-HB-002 


Page  11-127 


1.5.2  Prepare  Data  Collection  Plan  (continued) 


Parameters 
and  Limits 
(continued) 


Optional 

Practices 


The  data  collection  plan  (often  documented  in  a  variety  of  artifacts)  includes 

•  assignment  of  PAs  to  team  members 

•  summary  of  initial  objective  evidence  provided  by  the  organization 

•  identification  of  highest  priority  data  needs 

•  initial  allocation  of  data  needs  to  data-gathering  events 

•  identification  of  instruments  to  be  administered 

•  identification  of  participants  to  be  interviewed 

•  interview  schedule,  revised  to  include  more  detail 

•  identification  of  a  starter  set  of  interview  questions 

•  identification  of  documents  still  needed  (if  any) 

•  risks  associated  with  the  sufficiency  of  the  data  and  the  adequacy  of  the 
schedule 


Review  the  status  of  the  objective  evidence  database  with  members  of  the 
appraised  organization  to  elicit  additional  objective  evidence  or  to  expand  on 
the  evidence  available.  This  allows  the  appraisal  team  leader  to  validate  the 
data  collection  plan  to  some  extent. 


Continued  on  next  page 
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1.5.2  Prepare  Data  Collection  Plan  (continued) 


Implementation 

Guidance 


Sources  of  objective  evidence  include  instruments,  documents,  presentations, 
and  interviews  (see  process  2.1,  Examine  Objective  Evidence).  Objective 
evidence  is  differentiated  in  terms  of  different  types  of  PIIs  (direct  artifacts, 
indirect  artifacts,  and  affirmations),  as  described  in  activity  1.4.3,  Obtain 
Initial  Objective  Evidence.  A  combination  of  these  indicator  types  is  required 
for  corroboration  (see  activity  2.2.1,  Verify  Objective  Evidence).  The  data 
collection  status  is  continually  monitored  during  appraisal  activities  (see 
process  2.3,  Document  Objective  Evidence)  to  ensure  that  sufficient  data 
coverage  is  obtained.  These  are  all  key  considerations  that  should  be 
understood  and  accounted  for  in  the  generation  of  the  data  collection  plan. 

Multiple  types  of  interviews  can  be  used  to  obtain  face-to-face  affirmations 
(see  activity  2.1.4,  Examine  Objective  Evidence  from  Interviews): 

•  standard  structured  interviews  scheduled  in  advance  and  using  scripted 
questions 

•  on-call  interviews,  scheduled  in  advance  for  calendar  purposes,  but  held 
only  if  it  is  determined  they  are  necessary 

•  office  hours  interviews,  for  which  interviewees  are  notified  that  they  may 
need  to  be  available  as  a  contingency  during  scheduled  periods 

A  robust  data  collection  plan  will  plan  for  interviews  of  all  three  types.  Start 
with  a  full  set  of  scheduled  interviews  early  in  the  planning  phase,  and 
gradually  add/eliminate/modify  events  as  the  inventory  of  initial  objective 
evidence  indicates  the  need.  The  mini-teams  may  conduct  office  hours 
interviews,  even  during  team  training,  to  more  fully  populate  the  inventory  of 
objective  evidence  prior  to  the  start  of  the  on-site  data  collection  activities. 

Planning  for  document  reviews  should  include  organizational-,  project-,  and 
implementation-level  artifacts,  as  described  in  activity  2.1.3,  Examine 
Objective  Evidence  from  Documents. 

Ultimately,  the  appraisal  team  will  need  to  have  data  on  each  practice  in  the 
CMMI  model,  for  each  organizational  element  in  the  appraisal  scope.  For  PAs 
addressing  practices  implemented  at  the  project/program  level  (e.g..  Project 
Planning),  this  means  that  data  on  each  instantiation  of  the  practice  will  be 
collected.  For  PAs  addressing  practices  implemented  at  the  organization  level 
(e.g..  Organizational  Training),  only  one  instantiation  of  each  practice  may  be 
needed,  depending  on  the  way  the  organization  chooses  to  implement  such 
practices. 


Continued  on  next  page 
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1.5.2  Prepare  Data  Collection  Plan  (continued) 


Implementation 

Guidance 

(continued) 


The  results  of  the  analysis  of  initial  objective  evidence  are  used  to  determine 
which  practices  are  not  already  covered  with  objective  evidence.  Practices  for 
which  no  initial  objective  evidence  has  been  provided  should  be  identified  as 
high-risk  areas  for  the  team  to  address  immediately.  The  schedule  for  data 
collection  may  need  to  change  dramatically  if  the  team  is  unable  to  find 
relevant  data  for  these  areas  in  short  order.  In  the  case  of  practices  for  which 
data  are  available  in  the  initial  objective  evidence,  the  team  members  assigned 
to  the  PAs  plan  the  strategy  for  verifying  the  implementation  of  each  of  the 
practices  through  review  of  the  named  documents,  interviews  with  the  people 
who  play  the  named  roles,  or  other  data  collection  events.  Artifacts  used  to 
manage  data  collection  events  are  populated  with  the  current  understanding  of 
the  planned  data  collection  events,  as  follows: 

•  The  schedule  for  interviews  is  finalized,  so  participants  can  be  informed 
of  the  expectations  for  their  participation  as  interviewees. 

•  The  list  of  documents  on  hand  (or  accessible  electronically)  is  finalized, 
so  that  the  team  members  know  what  is  and  is  not  available  for  document 
review. 

•  A  preliminary  allocation  of  practices  to  be  covered  in  each  of  the 
scheduled  interviews  is  documented. 

•  A  list  of  needed  documents  (not  yet  available  to  the  team)  is  generated,  if 
there  are  any  known  needs  for  documents  at  this  point. 
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1.5.2  Prepare  Data  Collection  Plan 
GSS  and  CPM  Implementation  Guidance 

When  using  the  MDD  for  Government  source  selection  and  contract  process  monitoring,  the 
following  should  be  considered: 

Guidance  for  For  GSS  or  CPM,  a  data  collection  plan  may  differ  by  emphasizing  certain 

GSS  and  CPM  areas  of  concern.  For  example,  a  contract  may  require  intense  engineering 

activities.  Therefore,  engineering  PAs  may  need  to  be  more  thoroughly 
investigated.  However,  corroboration  rules  still  apply  for  all  practices  (see 
activity  2.2.1,  Verify  Objective  Evidence).  The  rules  for  data  collection  and 
corroboration  for  a  SCAMPI  ARC  Class  A  evaluation  are  stringent  and  may 
not  be  compatible  with  the  constraints  of  the  source  selection  (e.g.,  schedule). 
On  the  other  hand,  an  ARC  Class  A  evaluation  can  be  accomplished  even  if 
the  evaluation  scope  is  decreased  by  selective  inclusion  of  PAs.  If  source 
selection  constraints  are  present  that  don’t  allow  a  full  ARC  Class  A 
evaluation  of  all  maturity  level  2  and  3  PAs,  then  the  needs  of  the  program 
(risks)  should  drive  the  implementation  approach  regarding  whether  to  reduce 
PA  scope  or  verification  scope. 

Guidance  for  Source  selection  constraints  on  communications  with  offerors  may  affect  the 
GSS  data  collection  planning  process.  For  example,  review  of  the  data  collection 

plan  with  the  appraised  organization  may  not  be  possible.  Similarly, 
identification  of  interview  participants  may  not  be  possible  until  arriving  on 
site. 

The  finalization  of  the  data  collection  plan  may  drive  the  issuance  of  source 
selection  clarification  notices,  such  as  Evaluation  Notices,  Requests  for 
Clarification,  and  Points  for  Negotiation,  to  formally  request  the  additional 
data  required  prior  to  start  of  the  on-site  data  collection.  Evaluation  Notices 
may  also  be  used  to  issue  surveys,  instruments,  or  even  interview  questions  if 
appropriate. 
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1 .5.3  Replan  Data  Collection 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


The  data  collection  plan  is  updated  as  required  during  the  conduct  of  the 
readiness  review  or  during  the  appraisal  itself  as  objective  evidence  is  found, 
or  as  new  sources  of  information  are  uncovered.  The  activity  described  in  this 
section  refers  to  a  more  substantial  change  in  the  plan,  which  is  expected  to  be 
a  rare  occurrence  in  practice.  If  during  the  conduct  of  an  appraisal,  the  team 
discovers  that  their  assumptions  about  the  availability  of  objective  evidence 
are  substantially  incorrect,  the  appraisal  team  leader  may  renegotiate  the 
appraisal  plan  with  the  sponsor. 


•  Review  the  current  inventory  of  objective  evidence  and  determine  model 
practices  for  which  the  objective  evidence  is  inadequate  relative  to  the 
appraisal  plan. 

•  Revise  the  data  collection  plan  as  necessary  based  on  the  appraisal  status 
and  availability  of  objective  evidence. 

•  Renegotiate  the  appraisal  plan  with  the  sponsor  if  the  appraisal  cannot 
proceed  as  planned. 


This  activity  is  not  a  substitute  for  tactical  decisions  about  where  and  how  to 
find  objective  evidence.  The  intent  of  this  activity  is  to  respond  to  a  major  gap 
between  expected  data  and  actual  data. 

Major  gaps  between  expected  and  actual  data  may  occur,  for  example,  as  a 
result  of  the  following: 

•  inaccurate  assumptions  about  the  availability  of  objective  evidence 

•  content  of  artifacts  or  information  from  interviews  not  providing 
significant  amounts  of  the  information  required  and  other  sources  not 
being  planned 

•  unexpected  absence  of  multiple  key  interviewees 

•  unanticipated  delays  in  the  implementation  of  new  processes 

•  major  customer-driven  emergencies  for  one  or  more  of  the  sampled 
projects 


Continued  on  next  page 
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1.5.3  Replan  Data  Collection  (continued) 


Optional  Risk  analysis  can  be  conducted  during  early  planning  activities  to  establish 

Practices  thresholds  and  limits  for  the  amount  of  missing  objective  evidence  that  will 

trigger  this  activity.  This  enables  the  appr.  isal  team  leader  to  state,  in 
advance,  the  conditions  under  which  the  team  and  the  sponsor  must 
renegotiate  the  appraisal  plan. 

Contingency  planning  done  in  advance  to  identify  ways  of  overcoming  issues 
associated  with  missing  objective  evidence  could  include 

•  an  alternate  (fail-back)  schedule  for  the  appraisal 

•  staffing  to  conduct  a  “crash  data  collection”  activity 

•  reducing  the  scope  of  the  appraisal  (e.g.,  appraising  fewer  PAs,  limiting 
the  extent  of  the  organizational  unit  appraised) 


Implementation  This  activity  serves  as  a  “pressure  valve”  of  sorts  for  the  appraisal.  The 
Guidance  pressure  to  perform  the  appraisal  under  unrealistic  conditions  can  lead  to  a 

severe  degradation  in  the  quality  of  the  appraisal  outputs.  Carefully  planning 
for  contingencies  and  communicating  them  to  the  sponsor  help  to  protect  the 
standards  that  must  be  met  in  the  performance  of  an  appraisal.  Clearly 
documenting  the  data  collection  plan,  and  regularly  monitoring  the  availability 
of  data  compared  to  that  plan,  support  effective  risk  mitigation. 

When  this  activity  must  be  employed  to  recover  from  an  unrealistic 
expectation,  the  documentation  reflecting  the  assumptions  made  during 
planning,  as  well  as  concrete  facts  about  what  is  or  is  not  available,  are  used  to 
renegotiate  with  the  appraisal  sponsor.  This  is  one  of  the  reasons  why  a 
detailed  appraisal  plan,  with  the  sponsor’s  signature,  is  a  required  artifact  for 
the  conduct  of  a  SCAMPI  appraisal. 
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1 .5.3  Replan  Data  Collection 


GSS  and  CPM  Implementation  Guidance 

When  using  the  MDD  for  Government  source  selection  and  contract  process  monitoring,  the 
following  should  be  considered: 

Guidance  for  If  renegotiation  of  the  appraisal  plan  becomes  necessary  due  to  any  of  the 
GSS  and  CPM  aforementioned  circumstances  (e.g.,  lack  of  sufficient  objective  evidence, 
proceeding  in  a  discovery-based  appraisal  instead  of  a  verification-based 
appraisal),  the  source  selection  authority  may  decide  to  reduce  the  scope  of 
the  appraisal  because  of  time  constraints  and  focus  primarily  dii  the  areas  of 
emphasis  for  the  contract.  Note  discussion  earlier  in  the  Executive  Summary 
(“What  is  SCAMPI,”  and  “SCAMPI  Tailoring”)  regarding  SCAMPI  ARC 
Class  A  requirements.  Reducing  model  scope  and/or  requirements  for  ratings, 
etc.  could  preclude  accomplishing  a  benchmarking  SCAMPI  ARC  Class  A 
appraisal.  In  this  instance,  an  ARC  Class  B  or  C  appraisal  may  be  the  most 
viable  option  to  continue  with  the  appraisal  in  the  GSS  environment.  For 
CPM,  renegotiation  of  the  appraisal  plan  or  schedule  would  normally  be 
acceptable. 
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2.1  Examine  Objective  Evidence 


Purpose 

Entry  Criteria 

Inputs 


Activities 

Outputs 

Outcome 

Exit  Criteria 


Collect  information  about  the  practices  implemented  in  the  organization  and 
relate  the  resultant  data  to  the  reference  model.  Perform  the  activity  in 
accordance  with  the  data  collection  plan.  Take  corrective  actions  and  revise 
the  data  collection  plan  as  needed. 


•  Data  collection  has  been  planned. 

•  The  sponsor  has  approved  the  appraisal  plan. 

•  The  appraisal  team  is  trained  and  is  familiar  with  the  appraisal  plan. 

•  Participants  have  been  informed  about  the  appraisal  process  and  their 
roles  in  it. 


•  Appraisal  data 

-  initial  objective  evidence 
documents 

-  documented  practice  implementation  gaps,  if  any 

feedback  on  preliminary  findings  (if  that  point  in  the  timeline  has 
been  reached) 

•  Data  collection  plan 

appraisal  schedule 
interview  schedule 

-  document  list 

-  new  interview  questions 


2.1.1  Examine  Objective  Evidence  from  Instruments 

2.1.2  Examine  Objective  Evidence  from  Presentations 

2. 1 .3  Examine  Objective  Evidence  from  Documents 

2. 1 .4  Examine  Objective  Evidence  from  Interviews 


•  Updated  appraisal  data 

•  Updated  data  collection  plan 


After  the  final  iteration  of  this  process,  the  team  has  sufficient  data  to  create 
appraisal  findings  and  to  make  judgments  about  the  implementation  of 
practices,  as  well  as  the  satisfaction  of  specific  and  generic  goals. 


The  coverage  of  the  reference  model  and  the  organizational  scope  has  been 
achieved,  and  the  team  is  ready  to  produce  the  appraisal  outputs. 


Continued  on  next  page 
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2.1  Examine  Objective  Evidence  (continued) 


Key  Points 

Tools  and 
Techniques 

Metrics 

Verification  and 
Validation 

Records 

Tailoring 


The  efficient  collection  of  objective  evidence  results  from  carefully  creating 
and  executing  the  data  collection  plan.  Effective  contingency  planning  and  the 
use  of  work  aids  to  monitor  progress  are  key  points  to  consider.  The  team 
must  be  able  to  focus  on  examining  the  most  relevant  information  available, 
rather  than  be  distracted  by  a  mission  to  root  out  new  evidence. 


Wall  charts  and  other  visual  aids  are  often  used  to  display  the  results  of  data 
collection  activities.  Electronic  tools  are  prevalent  among  experienced  Lead 
Appraisers,  and  can  be  very  effective  for  continually  monitoring  and  updating 
the  inventory  of  objective  evidence. 


Tracking  the  actual  coverage  obtained,  as  compared  to  the  planned  coverage, 
in  each  data  collection  activity  facilitates  timely  corrective  actions  where  they 
are  needed.  The  most  critical  resource  during  an  appraisal  is  time.  Using  a 
timekeeper  during  data  collection  and  verification  activities  provides  feedback 
on  team  performance.  Recording  the  actual  duration  of  planned  events  helps 
the  team  in  taking  actions  to  recover  from  unexpected  events. 


The  appraisal  method  provides  detailed  verification  and  validation  procedures 
for  objective  evidence.  They  are  described  in  process  7,  Verify  and  Validate 
Objective  Evidence. 


Work  aids  used  to  record  and  track  the  progress  of  data  collection  activities 
are  retained  for  traceability  and  provide  an  important  input  to  a  final  report 
describing  the  appraisal,  if  the  sponsor  has  requested  a  final  report.  The 
duration  and  effort  required  for  specific  data  collection  events  can  be  recorded 
to  provide  useful  historical  data  for  planning  subsequent  appraisals. 


The  method  is  flexible  in  terms  of  the  use  of  customized  data  collection 
instruments,  presentations,  document  reviews,  and  interviews.  Specialized 
forms  of  these  data  collection  methods  can  be  constructed  to  meet  the 
objectives  of  the  appraisal.  For  example,  an  organization-specific 
questionnaire  could  be  used  that  contains  local  jargon  rather  than  a 
standardized  questionnaire.  Standardized  presentations  can  be  employed  to 
provide  the  team  with  an  “inbrief’  at  the  start  of  the  appraisal.  The  method 
also  provides  flexibility  in  choosing  the  number,  duration,  style,  and  make-up 
of  interview  sessions  within  specified  boundaries. 


Continued  on  next  page 
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2.1  Examine  Objective  Evidence  (continued) 


Interfaces 
with  Other 
Processes 


Summary  of 
Activities 


The  activities  that  provide  the  team  with  data  needed  to  produce  reliable 
appraisal  outputs  are  perhaps  the  most  visible  part  of  the  appraisal  process 
from  the  perspective  of  the  appraised  organization.  For  this  reason,  SCAMPI 
places  a  heavy  emphasis  on  methodically  planning  and  tracking  the  data 
collected  during  an  appraisal.  The  initial  objective  evidence  collected  early  in 
the  process  allows  team  members  to  analyze  the  state  of  information  available 
at  die  earliest  stages  of  the  appraisal  and  narrow  their  search  for  new 
information.  This  early  work  serves  to  facilitate  an  efficient  use  of  time.  An 
explicit  understanding  of  what  information  is  needed  and  how  that 
information  will  be  used  therefore  drives  the  activities  associated  with  this 
process. 


The  members  of  the  team  continually  manage  the  data  collected  previously 
and  target  new  data  collection  activities  to  fill  known  information  needs. 
Instruments  tend  to  be  used  early  in  the  appraisal  process,  and  often  provide 
leads  to  be  pursued  through  other  data  collection  activities,  in  addition  to 
affirmations  of  implemented  practices.  Presentations  are  sometimes  used  to 
provide  a  flexible  forum  where  members  of  the  organization  can  explain 
important  information  about  the  practices  implemented  in  the  organization. 
Documents  provide  the  most  explicit  and  lasting  representation  of  practice 
implementation  in  the  organization,  and  the  team  uses  them  to  understand  how 
practices  in  the  CMMI  model  are  implemented.  Finally,  interviews  are  used  as 
the  most  dynamic  data  collection  technique,  allowing  for  branching  among 
related  topics  and  the  explanation  of  contextual  information  that  affects  the 
implementation  of  practices  as  well  as  alternative  practices. 

The  appraisal  activities  conducted  for  each  of  these  data  collection  sources  are 
similar: 

•  Determine  if  the  information  obtained  is  acceptable  as  objective  evidence. 

•  Relate  the  objective  evidence  to  corresponding  practices  in  the  appraisal 
reference  model. 

•  Relate  the  objective  evidence  to  the  appropriate  part  of  the  appraised 
organizational  unit  (i.e.,  the  instantiation). 
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2.1  Examine  Objective  Evidence 
GSS  and  CPM  Implementation  Guidance 


No  additional  guidance. 
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2.1 .1  Examine  Objective  Evidence  from  Instruments 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Instruments  provided  by  the  organizational  unit  are  reviewed  to  obtain 
objective  evidence  reflecting  the  organization’s  implementation  of  model 
practices.  Instruments  include  questionnaires,  surveys,  and  other  written 
information  that  indicates  practice  implementation. 

This  activity  builds  upon  the  inventory  of  objective  evidence  that  was 
developed  during  appraisal  planning  and  preparation.  The  appraisal  team 
considers  the  information  contained  in  the  instruments  and  determines  if  it  is 
accurate,  consistent,  and  relevant  to  the  scope  of  the  reference  model  being 
examined. 

Objective  evidence  obtained  from  instruments,  and  from  other  sources,  is 
documented  in  process  2.3,  Document  Objective  Evidence,  and  verified  in 
process  2.2,  Verify  and  Validate  Objective  Evidence. 


•  Review  information  obtained  from  instruments  and  determine  if  it  is 
acceptable  as  objective  evidence. 

•  Determine  the  model  practices  corresponding  to  the  objective  evidence 
obtained  from  instruments. 

•  Determine  the  portions  of  the  organizational  unit  that  correspond  to  the 
objective  evidence  obtained  from  instruments. 


At  least  one  instrument  must  be  used  during  the  conduct  of  the  appraisal. 
(Refer  to  1.4.2,  Administer  Instruments,  for  a  description  of  instruments.) 


Continued  on  next  page 
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2.1 .1  Examine  Objective  Evidence  from  Instruments  (continued) 


Optional  Summaries  of  practice  implementation  data  (collected  via  instruments)  for  a 

Practices  group  of  proje  in  an  organization  may  be  useful  during  the  selection  of  the 

projects  used  to  represent  the  organizational  unit. 

•  The  use  of  legacy  processes  (versus  newly  deployed  processes)  can  be 
flagged  using  the  responses  to  these  instruments.  This  can  support  the 
inclusion  or  exclusion  of  projects  using  various  versions  of  the 
organization’s  set  of  standard  processes. 

•  This  can  also  help  flag  situations  where  projects  have  not  yet  reached  a 
particular  point  in  the  life  cycle,  allowing  the  appraisal  team  leader  to 
avoid  the  anomalous  situation  where  none  of  the  sampled  projects  has  yet 
reached  the  point  where  a  practice  under  investigation  has  been 
implemented. 

Create  and  administer  a  specialized  questionnaire  that  is  tailored  to  the 
characteristics  of  the  organization,  or  the  objectives  of  the  appraisal. 


Implementation  The  use  of  instruments  to  gather  written  information  from  members  of  the 
Guidance  organization  provides  a  relatively  low-cost  data  collection  technique,  when 

done  well.  Data  of  this  type  tend  to  be  most  useful  when  provided  early  in  the 
appraisal  conduct,  and  can  lead  to  valuable  insights  about  where  data  may  be 
sought  during  subsequent  data  collection  events. 

The  most  common  instrument  used  is  the  organization’s  PH  database,  which 
provides  traceability  of  reference  model  practices  to  the  processes  and  work 
products  implemented  within  the  organization.  Where  organizations  have  not 
yet  implemented  this  asset,  a  questionnaire  can  be  used  to  gather  closed-ended 
responses  and  comments  about  the  implementation  of  each  model  practice  in 
each  sampled  project  in  the  organizational  unit. 

It  is  also  the  responsibility  of  the  appraisal  team  leader  to  prevent  duplicate 
data  entry  on  multiple  instruments.  No  organization  should  be  asked  to 
provide  the  same  information  in  two  (or  more)  formats. 
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2.1 .1  Examine  Objective  Evidence  from  Instruments 


GSS  and  CPM  Implementation  Guidance 

When  using  the  MDD  for  Government  source  selection  and  contract  process  monitoring,  the 

following  should  be  considered: 

Guidance  for  Inclusion  of  instruments  in  Section  L  of  the  RFP  may  be  appropriate  and 

GSS  supportive  of  the  review  of  submitted  initial  data  and  formulation  of  the  data 

collection  plan. 

As  an  alternative,  they  could  be  issued  as  source  selection  clarification 
notices,  such  as  Evaluation  Notices,  Requests  for  Clarification,  and  Points  for 
Negotiation,  with  the  responses  due  prior  to  or  during  the  on-site  data 
collection. 

In  the  source  selection  environment,  it  is  prudent  to  take  a  more  conservative 
approach  in  determining  the  appropriateness  of  instruments.  This  is  because  of 
the  nature  of  how  these  instruments  are  completed.  In  a  supplier  selection,  the 
natural  tendency  of  the  respondent  in  a  development  organization  that  has 
been  asked  to  complete  a  publicly  available  document  is  to  “put  their  best  foot 
forward.”  This  same  general  rationale  applies  to  other  types  of  data  not 
directly  controlled  by  the  sponsoring  agency  (e.g.,  internal  assessment  results, 
past  contract  performance).  The  caution  to  apply  in  using  any  of  this  type  of 
data  is  understanding  its  applicability,  timeliness,  and  source.  That  is, 
responses  contained  in  instruments  still  need  to  be  verified  and  corroborated. 

The  responses  to  the  instruments  requested  may  play  a  role  in  evaluating  the 
initial  proposal  data  submittal  for  determining  the  responsiveness  of  the 
offerors.  The  clarity  and  completeness  of  the  responses  to  instruments  will 
also  drive  the  planning  and  scheduling  of  the  on-site  appraisal  both  in  terms  of 
selecting  areas  (PAs)  for  emphasis  in  verification  and  discovery  and 
identifying  offeror  participants  for  interviews. 

Guidance  for  No  additional  guidance. 

CPM 

♦ 
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2.1 .2  Examine  Objective  Evidence  from  Presentations 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Demonstrations  of  on-line  tools,  or  libraries  to  be  accessed  by  the  appraisal 
team,  are  often  the  best  way  for  members  of  the  team  to  find  the  data  and 
information  they  need.  The  history  of  process  improvement  in  the 
organization  or  the  status  of  current  improvement  projects  can  sometimes  be 
best  conveyed  to  the  appraisal  team  in  the  form  of  a  presentation.  While  the 
amount  of  data  to  be  collected  using  presentations  will  be  minimal,  the  ability 
to  receive  information  and  ask  questions  in  real  time  makes  this  a  valuable 
data  collection  technique. 

Objective  evidence  obtained  from  presentations,  and  from  other  sources,  is 
documented  in  process  2.3,  Document  Objective  Evidence,  and  verified  in 
process  2.2,  Verify  and  Validate  Objective  Evidence. 


•  Receive  presentations,  if  applicable,  from  the  organizational  unit. 

•  Review  information  obtained  from  presentations  and  determine  if  it  is 
acceptable  as  objective  evidence. 

•  Determine  the  model  practices  corresponding  to  the  objective  evidence 
obtained  from  presentations. 

•  Determine  the  portions  of  the  organizational  unit  that  correspond  to  the 
objective  evidence  obtained  from  presentations. 


There  is  no  requirement  for  one  or  more  presentations  to  be  included  in  the 
data  collection  plan.  The  team  must  permit  presentations  of  information  by 
knowledgeable  members  of  the  organization.  Presentations  may  or  may  not  be 
“required”  by  the  team,  depending  on  the  appraisal  usage  mode  and  the 
appraisal  objectives. 

It  is  not  necessary  that  all  team  members  be  present  at  every  presentation, 
though  it  may  be  advantageous.  A  minimum  of  two  team  members  must  be 
present  in  order  to  consider  any  presentation  a  valid  data  collection  session. 

Team  members  take  notes  during  presentations  to  document  information  for 
-later  use,  as  described  in  activity  2.3.1,  Take/Review/Tag  Notes. 


Continued  on  next  page 
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2.1 .2  Examine  Objective  Evidence  from  Presentations  (continued) 


Optional 

Practices 


Implementation 

Guidance 


Page  11-148 


Allow  the  organization  to  provide  presentations  or  demonstrations  of  tools,  as 
a  means  of  providing  objective  evidence  about  the  implementation  of  model 
practices. 

Establish  a  standardized  boilerplate  for  the  organizational  unit,  or  projects 
within  the  organizational  unit,  to  use  in  orienting  the  appraisal  team. 


Presentations  about  the  history  of  process  improvement  in  an  organization  can 
be  very  revealing,  and  can  help  to  shape  the  emphasis  for  further  data 
collection. 

Demonstrations  of  tools  supporting  the  process  infrastructure  are  sometimes 
the  most  convenient  means  of  communicating  objective  evidence.  Tools  that 
are  commonly  demonstrated  include 

•  requirements  management  and  traceability  tools 

•  configuration  management  library 

•  metrics  database 

•  process  asset  library  and  tools 

•  process-related  Web  pages 

•  computer-based  training  courses  or  training  repositories 

•  risk  management  databases 

A  configuration  management  library  often  embodies  the  process  by  which 
engineers  manage  configurations.  These  engineers  may  take  for  granted  that 
certain  standards  are  enforced  through  the  tool  and  be  unable  to  explain  what 
those  standards  are  in  the  abstract. 

An  organization’s  metrics  database  can  often  embody  the  analytical 
techniques  in  use,  as  well  as  the  communication  channels  that  are  supported 
across  the  organizational  unit. 


♦ 
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2.1 .2  Examine  Objective  Evidence  from  Presentations 


GSS  and  CPM  Implementation  Guidance 

No  additional  guidance. 


2.1 .3  Examine  Objective  Evidence  from  Documents 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


A  substantial  portion  of  the  data  used  by  appraisal  team  members  is  derived 
from  documents  they  review.  Most  of  the  direct  artifacts  used  as  indicators  of 
practice  implementation  are  documents.  Document  review  is  an  effective 
means  to  gain  detailed  insight  about  the  practices  in  use  in  the  organization. 
However,  without  a  clear  focus  on  the  data  being  sought,  document  review 
can  consume  a  great  deal  of  time  as  team  members  sometimes  attempt  to  read 
everything  in  hopes  that  something  useful  will  be  discovered. 

Objective  evidence  obtained  from  documents,  and  from  other  sources,  is 
documented  in  process  2.3,  Document  Objective  Evidence,  and  verified  in 
process  2.2,  Verify  and  Validate  Objective  Evidence. 


•  Establish  and  maintain  a  catalogue  of  documents  used  as  a  source  of 
Objective  Evidence. 

•  Review  information  obtained  from  documents  and  determine  if  it  is 
acceptable  as  objective  evidence. 

•  Determine  the  model  practices  corresponding  to  the  objective  evidence 
obtained  from  documents. 

•  Determine  the  portions  of  the  organizational  unit  that  correspond  to  the 
objective  evidence  obtained  from  documents. 


All  SCAMPI  appraisals  must  use  documents  as  a  source  of  information  on  the 
extent  to  which  practices  have  been  implemented  in  the  organizational  unit 
and  within  the  sampled  projects. 

The  catalogue  should  be  sufficient  to  summarize  the  documentation  objective 
evidence  used  as  a  basis  for  appraisal  ratings  generated,  as  required  by  the 
Appraisal  Record  described  in  activity  3.2.2,  Generate  Appraisal  Record. 
Much  of  the  catalogue  contents  can  be  obtained  from  the  mapping  data  or 
instruments  obtained  from  the  organizational  unit,  such  as  the  PH  database,  or 
questionnaires.  The  catalogue  can  be  used  to  maintain  a  list  of  documents 
reviewed  or  additional  documentation  requested  from  the  organizational  unit. 


Continued  on  next  page 
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2.1 .3  Examine  Objective  Evidence  from  Documents  (continued) 


Optional 

Practices 


Implementation 

Guidance 


Implementation 

Guidance 

Three  Levels  of 
Documents 
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For  organizations  with  substantial  intranets  containing  Web-based  document 
libraries,  a  member  of  the  organization  familiar  with  the  document  library 
should  provide  a  demonstration  of  the  Web-based  tools.  Links  to  other 
documents  and  other  features  of  the  Web-based  document  library  must  be 
tested  prior  to  the  team’s  use  during  the  appraisal. 


One  or  more  team  members  will  seek  data  for  every  practice  in  the  model 
scope  of  the  appraisal  through  document  review.  This  does  not  require  a 
document  for  every  practice,  as  any  given  document  is  likely  to  provide  data 
relevant  to  multiple  practices.  To  the  greatest  extent  possible,  the  location  of 
documented  evidence  relating  to  every  practice  should  be  recorded  in  advance 
of  the  team’s  arrival  at  the  site  where  the  appraisal  will  occur.  Organizations 
with  established  improvement  infrastructures  typically  maintain  this  type  of 
information  in  order  to  track  their  improvement  efforts  against  the  model. 
Where  this  information  is  incomplete,  the  team  will  be  forced  to  discover  the 
linkages  between  the  CMMI  model  and  the  organization’s  implemented 
practices,  and  will  therefore  require  more  time  to  perform  the  appraisal. 


Documents  reviewed  during  an  appraisal  can  be  classified  into  three  different 
levels:  organization,  project,  and  implementation. 

By  providing  further  insight  into  the  policies  and  procedures  that  guide  the 
organization’s  processes,  organization-level  documents  sometimes  help  the 
team  to  eliminate  the  need  for  a  question  during  an  interview  or  sharpen  the 
focus  for  a  question.  Review  of  these  documents  provides  a  context  for 
understanding  the  expectations  placed  on  projects  within  the  organization. 

Through  the  review  of  project-level  documents,  team  members  gain  further 
insight  into  each  scheduled  interviewee’s  role  in  the  project  they  support  as 
well  as  the  terminology  generally  accepted  within  the  organization  or  project. 
This  may  lead  to  the  refinement  or  modification  of  interview  questions. 

The  team  typically  reviews  implementation-level  documents  to  validate 
information  gathered  from  other  sources,  such  as  interviews  or  higher-level 
documents.  Documents  on  this  level  provide  an  audit  trail  of  the  processes 
used  and  the  work  performed.  The  review  of  these  documents  frequently 
provides  verification  of  practices  found  in  organization-  and  project-level 
documents. 
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GSS  and  CPM  implementation  Guidance 

No  additional  guidance. 
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2.1 .4  Examine  Objective  Evidence  from  Interviews 


Activity 

Description 


Required 

Practices 


Interviews  are  used  to  obtain  face-to-face  affirmations  relating  to  the 
implementation  of  processes  at  the  organizational  and  project  levels. 
Interviews  are  held  with  managers  and  practitioners  responsible  for  the  work 
being  performed.  The  appraisal  team  uses  interviews  to  understand  how  the 
processes  are  implemented  and  to  probe  areas  where  additional  coverage  of 
model  practices  is  needed. 

Interviews  are  a  required  and  necessary  component  of  a  SCAMPI  appraisal,  in 
all  usage  modes.  The  criteria  for  the  amount  of  face-to-face  affirmation 
objective  evidence  that  must  be  collected  are  described  in  activity  2.2.1, 
Verify  Objective  Evidence.  This  drives  the  development  of  the  initial 
interviewing  strategy  in  the  data  collection  plan  described  in  activity  1.5.2, 
Prepare  Data  Collection  Plan.  A  variety  of  interviewing  techniques  are 
available,  and  the  appraisal  team  leader  works  with  the  team  to  schedule  the 
most  appropriate  interview  types  for  the  situation. 

As  objective  evidence  is  gathered  throughout  the  appraisal,  the  data  collection 
plan  is  revised  as  necessary.  By  using  focused  investigation  techniques,  the 
need  for  interviews  may  be  either  increased  or  diminished,  as  long  as  the 
criteria  for  face-to-face  affirmations  are  satisfied. 

Objective  evidence  obtained  from  interviews,  and  from  other  sources,  is 
documented  in  process  2.3,  Document  Objective  Evidence,  and  verified  in 
process  2.2,  Verify  and  Validate  Objective  Evidence. 


•  Refine  the  data  collection  plan  to  determine  the  objective  evidence  that 
must  be  obtained  from  interview  participants. 

•  Review  information  obtained  from  interviews  and  determine  if  it  is 
acceptable  as  objective  evidence. 

•  Determine  the  model  practices  corresponding  to  the  objective  evidence 
obtained  from  interviews. 

•  Determine  the  portions  of  the  organizational  unit  that  correspond  to  the 
objective  evidence  obtained  from  interviews. 


Continued  on  next  page 
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2.1 .4  Examine  Objective  Evidence  from  Interviews  (continued) 


Parameters 
and  Limits 


Optional 

Practices 


All  SCAMPI  appraisals  must  use  interviews  as  a  source  of  information  on  the 
extent  to  which  practices  have  been  implemented  in  the  organizational  unit 
and  within  the  sampled  projects. 

All  interviews  must  include  at  least  two  members  of  the  appraisal  team 
designated  by  the  appraisal  team  leader. 

Full  coverage  of  the  CMMI  model,  the  organizational  unit,  and  the 
organization’s  life  cycle(s)  must  be  achieved  with  the  objective  evidence 
considered  by  the  team.  Therefore  the  pool  of  potential  interviewees  must 
cover  all  elements  of  the  process  in  use  in  the  organizational  unit. 

Project  and/or  program  management  personnel  are  typically  interviewed 
individually,  or  grouped  according  to  project.  The  focus  of  the  discussion  in 
these  interviews  will  therefore  be  scoped  to  a  particular  project,  rather  than 
across  the  sampled  projects. 

Functional  Area  Representatives  (FARs)  are  typically  interviewed  in  a  group, 
sampling  across  the  projects  within  the  organizational  unit.  The  focus  of  the 
discussion  in  these  interviews  will  therefore  be  scoped  to  a  particular  set  of 
practices,  used  across  the  projects. 

The  rules  of  confidentiality  and  the  expected  use  of  appraisal  data  must  be 
communicated  to  every  interviewee. 


Request  that  interviewees  bring  a  document  or  other  artifact  with  them  to  their 
interviews  for  a  “show-and-tell”  style  interview. 

Use  video/teleconference  technology  to  conduct  interviews  at  a  distance. 
Appraisers  are  cautioned  not  to  rely  too  heavily  on  this  method.  If  substantial 
portions  of  the  interview  data  are  gathered  using  this  technology,  it  may  tend 
to  limit  the  amount  of  information  collected. 


Continued  on  next  page 
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2.1 .4  Examine  Objective  Evidence  from  Interviews  (continued) 


implementation 

Guidance 


Interviews  provide  the  most  flexible  source  of  detailed  data.  Face-to-face 
interaction  with  people  who  enact  the  practices  being  investigated  allows  the 
team  to  seek  detailed  information  and  to  understand  the  interrelationships 
among  various  practices.  Detailed  information  to  address  specific  data 
collection  needs  can  be  sought  and  verified  in  real  time. 

It  is  important  to  avoid  sampling  interviewees  for  a  session  such  that  two 
people  in  the  same  reporting  chain  (e.g.,  a  superior  and  one  of  his  or  her  direct 
reports)  are  in  the  same  interview  session.  This  applies  to  members  of  the 
appraisal  team  as  well.  People  who  have  this  type  of  relationship  with  one 
another  may  be  uncomfortable  with  the  expectation  for  them  to  be  completely 
candid  during  the  interview. 

Samples  of  interviewees  are  typically  grouped  into  categories  that  roughly 
correspond  to  life-cycle  phases,  engineering  disciplines,  organizational 
groupings,  and/or  PA  affinities.  As  stated  previously,  interviews  of 
project/program  management  personnel  are  typically  grouped  by  project, 
while  FARs  sampled  for  a  given  interview  come  from  across  the 
organizational  unit. 

There  are  three  basic  forms  of  interviews  used  in  SCAMPI.  They  are 
described  below. 


Continued  on  next  page 
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Guidance 
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Implementation 
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On-Call 
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The  most  structured  approach  is  the  standard  interview,  which  is  scheduled  in 
advance  and  employs  a  series  of  scripted  questions.  Each  standard  interview 
typically  involves  interviewees  with  similar  responsibilities  in  the 
organization  (e.g.,  Quality  Assurance  personnel,  Systems  Engineers,  or 
Middle  Managers).  The  schedule  and  location  of  each  interview  session  is 
communicated  to  the  interviewees  well  in  advance.  Questions  intended  to 
elicit  data  about  particular  practices  are  prepared  and  reviewed  in  advance, 
and  the  team  follows  a  defined  process  for  conducting  the  session.  The  entire 
team  is  present  for  these  interviews.  Responsibility  for  tracking  the  coverage 
of  individual  PAs  is  typically  assigned  to  team  members.  A  single  questioner 
may  lead  the  interview,  with  the  rest  of  the  team  listening  and  taking  notes,  or 
the  responsibility  for  asking  questions  may  be  distributed  among  the  team 
members.  In  any  case,  it  is  expected  that  all  team  members  who  are  not  asking 
questions  listen  and  take  notes  for  all  questions. 

A  set  of  planned  interviews  will  be  defined  during  appraisal  planning.  As  the 
appraisal  progresses  and  the  objective  evidence  accumulates,  the  team  may 
find  it  convenient  to  cancel  one  or  more  of  these  interviews  to  use  the  time  for 
other  activities.  Such  changes  in  the  data  collection  plan  are  made  in  a  way 
that  does  not  violate  the  coverage  criteria  described  in  process  2.2,  Verify  and 
Validate  Objective  Evidence. 


A  more  flexible  approach  to  scheduling  interviews  is  available  in  the  form  of 
on-call  interviews,  a  variant  of  the  standard  interview.  Prospective 
interviewees  are  identified  and  notified  in  advance,  just  as  described  above. 
However,  the  interviews  are  only  held  if  team  members  decide  that  there  is  a 
need  and  that  the  time  will  be  well  spent.  The  prospective  interviewees  are 
therefore  asked  to  block  a  period  of  time  for  such  a  contingency,  and  are 
informed  the  day  before  the  scheduled  time  as  to  whether  or  not  the  interview 
session  will  actually  happen.  These  interviews  need  not  include  the  entire 
appraisal  team,  thus  permitting  parallel  sessions  with  different  interviewees. 
However,  at  least  two  members  of  the  appraisal  team  (selected  by  the 
appraisal  team  leader)  must  participate. 


Continued  on  next  page 


CMU/SEI-2002-HB-002 
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Implementation 

Guidance 

Office  Hours 
Interviews 


Finally,  office  hours  interviews  represent  an  agreement  for  availability  that 
permits  pairs  of  team  members  to  visit  interviewees  at  their  desks,  cubicles,  or 
offices.  As  with  the  on-call  interviews,  the  prospective  interviewees  block  a 
specific  time  period  to  be  available  on  a  contingency  basis.  It  is  expected  that 
most  prospective  interviewees  will  be  able  to  continue  with  their  daily  work 
and  accommodate  an  interruption  if  the  team  needs  to  speak  with  them.  Here 
again,  only  if  specific  data  needs  are  identified  will  the  interview  occur.  The 
interviewees  should  be  informed  that  they  may  receive  oply  limited  advanced 
notice  for  these  interviews,  although  confirming  the  interview  at  least  a  day  in 
advance  is  a  courtesy  that  should  be  offered  whenever  possible. 
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2.1 .4  Examine  Objective  Evidence  from  Interviews 


GSS  and  CPM  Implementation  Guidance 

Guidance  for  Because  of  the  conservative  nature  of  the  GSS  process  and  the  controlled 

GSS  and  CPM  nature  of  the  communications  between  the  offerors  and  the  acquisition 
organization,  there  may  be  some  constraints  imposed  in  the  types  of 
interviews  allowed.  For  example,  group-on-one  interviews  may  not  be 
allowed  because  some  source  selection  organizations  may  not  consider  that 
the  interview  responses  would  represent  a  coordinated  position  of  the  offeror 
in  a  formal  sense.  The  issue  of  discussions  being  opened  and  the  impact  on 
interviews  needs  to  be  known  for  GSS.  Some  GSS  organizations/programs 
may  require  that  a  Contracting  Officer  be  present  during  the  interviews,  which 
may  constrain  the  types  of  interviews  available  for  practicable  use.  For 
example,  simultaneous  group-on-one  interviews  may  not  be  supported  by  one 
Contracting  Officer.  In  that  case,  group  interviews  may  be  more  appropriate. 
Focus  group  interviews  can  be  used  to  validate  preliminary  findings  with  the 
appraised  entity  without  doing  formal  presentations.  Make  sure  that  the  GSS 
environment  you  are  operating  in  supports  your  anticipated  method  for 
validating  preliminary  findings.  For  example,  some  source  selections  may  not 
allow,  or  may  place  constraints  on,  the  validation  of  preliminary  findings  as 
required  for  a  SCAMPI  ARC  Class  A  evaluation. 

The  CPM  environment  will  generally  have  a  more  flexible  approach  to 
interviews  due  to  the  circumstance  of  an  already  existing  contract  with  no 
competitors  involved.  However,  where  incentives  or  process  performance 
issues  are  critical,  appropriate  attention  to  data  collection  plans  and  the  types 
of  interviews  is  necessary  and  important  to  ensure  the  credibility  of  the  results 
obtained  from  the  SCAMPI  method. 
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2.2  Verify  and  Validate  Objective  Evidence 


Purpose 


Entry  Criteria 


Inputs 


Activities 


Outputs 


Outcome 


Exit  Criteria 


Verify  the  implementation  of  the  organization’s  practices  for  each 
instantiation,  and  validate  the  preliminary  findings,  describing  gaps  in  the 
implementation  of  model  practices.  Each  implementation  of  each  practice  is 
verified  so  that  it  may  be  compared  to  the  practices  of  the  CMMI  model,  and 
the  team  characterizes  the  extent  to  which  the  practices  in  the  model  are 
implemented.  Gaps  in  practice  implementation  are  captured  and  validated 
with  members  of  the  organization.  Exemplary  implementations  of  model 
practices  may  be  highlighted  as  strengths  to  be  included  in  appraisal  outputs. 


Objective  evidence  has  been  collected  about  the  implementation  of  practices 
in  the  organization.  Gaps  in  the  implementation  of  model  practices  have  been 
identified,  and  the  team  is  ready  to  characterize  the  extent  to  which  model 
practices  (or  acceptable  alternatives  to  those  practices)  have  been 
implemented.  Descriptions  of  practice  implementation  gaps  at  the  level  of  the 
organizational  unit  have  been  crafted  for  validation. 


•  Appraisal  plan,  with  schedule  and  participants  for  data  validation  activities 

•  Data  on  practice  implementation,  and  strength/weakness  statements 

•  Data  collection  plan,  specifying  any  additional  information  needed 


2.2. 1  Verify  Objective  Evidence 

2.2.2  Characterize  Implementation  of  Model  Practices 

2.2.3  Validate  Practice  Implementation  Gaps 


•  Updated  appraisal  data 

notes 

strength/weakness  statements 
annotated  worksheets 

•  Updated  appraisal  artifacts 

preliminary  findings 
revised  data  collection  plan 
requests  for  additional  data 


The  team’s  confidence  in  the  material'  that  will  form  the  basis  for  appraisal 
outputs  is  increased,  and  the  process  of  transferring  ownership  of  these  results 
has  been  started.  Any  critical  deficiencies  in  the  data  on  hand  have  been 
identified  and  actions  to  resolve  these  issues  have  been  initiated. 


The  team  has  recorded  data  on  the  implementation  of  practices  in  the 
organization,  and  characterized  the  extent  to  which  practices  in  the  model  are 
implemented.  In  addition,  strength  and  weakness  statements  have  been 
validated  with  members  of  the  organization  who  provided  appraisal  data. 


Continued  on  next  page 
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2.2  Verify  and  Validate  Objective  Evidence  (continued) 


Key  Points 


Tools  and 
Techniques 


Metrics 


Verification  and 
Validation 


Records 


Tailoring 


This  activity  spans  a  number  of  distinct  events  in  the  appraisal  method  that 
together  accomplish  the  same  goal — ensuring  the  validity  of  the  appraisal  data 
and  associated  outputs.  Managing  the  interaction  with  people  outside  of  the 
team  is  a  vitally  important  process  to  ensure  that  the  results  will  be  accurate. 


Facilitation  techniques  to  guide  the  team  through  difficult  decisions  are 
important  during  this  activity  (as  they  are  during  the  Rating  activity  as  well). 
Techniques  to  enhance  the  credibility  of  the  preliminary  findings  are  also 
important.  Using  a  flip  chart  or  note-taker  during  the  presentation  of 
preliminary  findings  is  often  effective  for  instilling  confidence  among 
audience  members. 


Planned  versus  actual  effort  expended  for  this  activity  (as  with  all  activities) 
will  assist  in  monitoring  progress  as  well  as  planning  subsequent  appraisals. 
Gauging  the  level  of  acceptance  for  preliminary  findings  can  be  facilitated  by 
computing  the  percentage  of  findings  adjusted  based  on  feedback,  then 
comparing  this  value  with  past  experience. 


The  attendees  of  preliminary  findings  presentations  are  likely  to  express 
agreement  and/or  discuss  issues  with  the  data  being  validated.  The  appraisal 
team  leader  needs  to  ensure  active  participation  in  these  activities  as  a  way  of 
verifying  that  the  verification  and  validation  process  is  working  as  intended. 
The  actions  taken  following  the  appraisal  will  provide  feedback  to  help 
validate  that  this  activity  was  successful. 


Characterizations  of  practice  implementation,  strength/weakness  statements 
and  changes  made  based  on  feedback  will  be  recorded  for  subsequent  use  by 
the  team. 


Validating  data  is  required,  but  a  variety  of  choices  for  orchestrating  this 
process  are  available.  The  most  common  approach  is  the  preliminary  findings 
presentation.  The  use  of  an  instrument  or  a  more  targeted  focus-group 
approach  to  validate  statements  of  practice  implementation  gaps  is  permitted. 
Also,  the  relative  emphasis  of  mini-team-based  verification  and  verification 
carried  out  by  the  team  as  a  whole  can  be  adjusted  to  meet  the  skills  and 
preferences  of  the  team  at  hand. 
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2.2  Verify  and  Validate  Objective  Evidence  (continued) 


Interfaces 
with  Other 
Processes 


Summary  of 
Activities 


During  the  conduct  of  an  appraisal,  the  team  must  gather  and  analyze  a  great 
deal  of  detailed  information.  Processes  described  earlier  in  this  document 
clarify  how  data  are  gathered  and  examined.  The  process  described  here  is 
focused  on  understanding  the  information  revealed  by  the  data.  The  processes 
described  after  this  one  are  focused  on  carefully  recording  important 
information  and  making  reliable  and  valid  rating  judgments  based  on  the 
verified  and  validated  data. 


The  initial  objective  evidence  provided  by  the  organization  is  used  to 
understand  how  practices  are  intended  to  be  implemented.  Members  of  the 
appraisal  team  then  seek  information  to  confirm  that  the  intended  practices  are 
indeed  implemented.  This  first  validation  activity  (2.2.1)  may  reveal  gaps  in 
the  actual  implementation  that  are  not  apparent  in  the  initial  objective 
evidence  provided  by  the  organization.  The  next  verification  activity  (2.2.2) 
then  compares  the  implemented  practices  to  the  practices  in  the  CMMI  model. 
This  activity  may  also  reveal  gaps  in  the  implementation(s)  that  will  later  bear 
on  the  ratings  assigned  by  the  team.  Standard  characterizations  to  capture  the 
extent  of  practice  implementation,  first  at  the  project  level  and  then  at  the 
organizational  unit  level,  are  recorded  by  the  team,  along  with  descriptions  of 
gaps  in  implementation.  When  team  members  have  achieved  their  planned 
coverage  of  data  collection,  the  descriptions  of  gaps  are  validated  with  the 
members  of  the  organization.  This  final  activity  prior  to  rating  allows  team 
members  to  build  confidence  that  their  investigation  has  been  thorough,  and 
the  members  of  the  organization  are  provided  with  an  opportunity  to  correct 
any  perceived  errors  in  the  appraisal  data. 


♦ 
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2.2  Verify  and  Validate  Objective  Evidence 


GSS  and  CPM  Implementation  Guidance 


No  additional  guidance. 
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2.2.1  Verify  Objective  Evidence 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


The  appraisal  team  must  establish  a  clear  understanding  of  the  practices 
implemented  in  the  organization.  Typically,  the  organization  provides  a  set  of 
objective  evidence  at  the  beginning  of  the  appraisal  process,  and  the  team  sets 
out  to  verify  the  instances  where  those  practices  are  implemented.  For 
practices  reflecting  project-level  activities,  the  team  must  observe  that  each 
selected  project  in  the  organizational  unit  has  evidence  of  implementation.  For 
practices  reflecting  organization-level  activities,  the  team  must  understand  the 
organization-level  implementation  as  well  as  any  activities  involving  the 
projects  that  indicate  the  implementation  of  the  practice. 


•  Verify  the  appropriateness  of  direct  artifacts  provided  by  each 
instantiation  for  practices  within  the  model  scope  of  the  appraisal. 

•  Verify  the  appropriateness  of  indirect  artifacts  provided  by  each 
instantiation  for  practices  within  the  model  scope  of  the  appraisal. 

•  Verify  the  appropriateness  of  affirmations  provided  by  each  instantiation 
for  practices  within  the  model  scope  of  the  appraisal. 

•  Verify  that  the  implementation  of  each  model  practice  is  supported  by 
direct  artifacts  for  each  instantiation,  and  corroborated  by  indirect  artifacts 
or  affirmations. 

•  Obtain  face-to-face  affirmations  for  (1)  at  least  one  instantiation  for  each 
model  practice  in  the  scope  of  the  appraisal,  or  (2)  at  least  50%  of  the 
practices  corresponding  to  each  specific  and  generic  goal  for  each 
instantiation. 

•  Generate  statements  describing  gaps  in  the  organizational  unit’s  implemented 
practices  relative  to  practices  defined  in  the  reference  model. 


For  practices  implemented  at  the  project  level,  direct  and  indirect  indicators  of 
practice  implementation  must  be  examined  for  every  project  sampled  to 
represent  the  organizational  unit  being  appraised. 

For  practices  implemented  at  the  organization  level,  direct  and  indirect 
indicators  of  practice  implementation  are  examined  in  reference  to  the 
organizational  unit  within  the  scope  of  the  appraisal,  and  not  necessarily  for 
each  project  sampled.  Aspects  of  the  practice  that  are  implemented  at  the 
project  level  must  be  investigated  for  every  project  sampled  to  represent  the 
organizational  unit. 

One  or  more  direct  artifacts  will  be  needed  to  verify  implementation  of  each 
model  practice.  Indirect  indicators  can  include  either  artifacts  or  affirmations. 
A  description  of  these  indicator  types  is  contained  in  activity  1.4.3,  Obtain 
Initial  Objective  Evidence. 

Coverage  criteria  for  face-to-face  affirmations  are  focused  at  the  goal  and 
organizational  unit  level. 


Continued  on  next  page 


CMU/SEI-2002-HB-002 


Page  11-169 


2.2.1  Verify  Objective  Evidence  (continued) 


Optional  At  the  discretion  of  the  appraisal  team  leader,  verific  ,tion  of  practices  at  the 

Practices  instantiation  level  may  be  carried  out  solely  by  the  :uini-teams.  Team-wide 

review  and  consensus  on  practice  implementation  can  then  focus  on  the 
aggregate-level  characterizations. 

At  the  discretion  of  the  appraisal  team  leader,  the  verification  of  practice 
implementation  at  the  project  level  can  be  reviewed  for  consensus  by  the 
entire  team.  Each  mini-team  provides  an  overview  of  practice  implementation 
indicators  for  each  project  sampled  to  represent  the  organizational  unit. 

A  mix  of  the  two  strategies  above  can  be  used,  selectively  reviewing  targeted 
PAs  in  different  ways,  or  gradually  changing  from  one  strategy  to  the  other  as 
the  team  gains  familiarity  with  the  data  and  the  process. 


Implementation  The  typical  work  products  listed  in  CMMI  models  provide  examples  of 
Guidance  artifacts  that  can  be  used  as  indicators  of  practice  implementation.  However, 

the  model  does  not  distinguish  between  direct  and  indirect  artifacts,  and  these 
are  examples  only  and  are  not  required;  alternatives  can  be  used  for  both 
direct  and  indirect  artifacts. 

Typically,  much  of  the  objective  evidence  required  to  perform  this  verification 
is  provided  in  advance  of  the  on-site  period.  The  primary  focus  of  data 
collection  is  to  permit  the  team  to  verify  that  the  intended  practices  are 
implemented  across  the  organizational  unit.  Where  the  implemented  practices 
differ  from  the  intended  practices,  the  objective  evidence  provided  at  the  start 
of  the  appraisal  process  is  annotated  to  more  accurately  reflect  the 
implemented  process  in  the  organization.  These  annotations  are  typically 
statements  describing  a  gap  in  the  implementation  of  a  model  practice,  some 
of  which  will  eventually  become  findings. 

Where  gaps  exist  in  the  objective  evidence  provided  in  advance,  the  appraisal 
team  is  forced  to  undertake  data  collection  activities  to  populate  the  data  set 
from  scratch.  An  organization  that  has  a  substantial  process  improvement 
infrastructure  in  place  is  expected  to  have  documented  its  implementation  of 
the  model  in  detail.  For  organizations  with  relatively  little  experience  using 
CMMI,  the  cost  of  this  discovery  process  may  be  so  great  that  undertaking  an 
ARC  Class  A  appraisal,  such  as  SCAMPI,  is  not  cost-effective.  For  such 
organizations,  an  ARC  Class  B  appraisal  may  be  more  appropriate. 

Only  after  team  members  have  a  clear  understanding  of  the  implemented 
practices  can  they  compare  them  to  the  model  to  characterize  the  extent  to 
which  the  organization  implements  the  practices  in  the  model  or  acceptable 
alternatives.  It  is  expected  that  artifacts  that  result  from  the  performance  of  the 
practice  will  be  available  for  viewing  by  the  team.  These  artifacts,  as  well  as 
face-to-face  interactions  with  members  of  the  organization  enacting  the 
practice,  help  to  verify  that  the  practice  was  enacted  as  the  maintainers  of  the 
organizational  process  intended. 


♦ 
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2.2.1  Verify  Objective  Evidence 


GSS  and  CPM  Implementation  Guidance 

No  additional  guidance. 
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2.2.2  Characterize  Implementation  of  Model  Practices 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Once  a  critical  mass  of  evidence  on  practice  implementation  has  been  verified, 
the  team  (or  mini-team)  turns  to  characterizing  the  implementation  of  model 
practices.  For  each  practice  in  the  model  included  in  the  selected  scope,  and 
each  instance  of  expected  use,  the  team  will  document  a  characterization  of 
the  extent  to  which  the  model  practice  (or  an  acceptable  alternative)  has  been 
implemented.  These  project-level  characterizations  are  then  aggregated  to  the 
organizational  unit  level. 

Characterizations  of  practice  implementation  are  Used  as  a  means  to  focus 
appraisal  team  effort  on  areas  where  professional  judgment  is  needed,  and  to 
aid  in  reaching  team  consensus  on  the  extent  to  which  practices  are 
implemented. 


•  Characterize,  for  each  instantiation,  the  extent  to  which  reference  model 
practices  are  implemented. 

•  Aggregate  practice  implementation  characterization  values  from  the 
instantiation  level  to  the  organizational  unit  level. 


The  following  table  summarizes  rules  for  characterizing  instantiation-level 
implementations  of  practices.  Consensus  of  at  least  a  subset  of  appraisal  team 
members  (e.g.,  mini-team  members)  is  necessary  for  instantiation-level 
characterizations. 


Label 

Meaning 

Fully  Implemented  (FI) 

•The  direct  artifact  is  present  and  judged  to  be 
appropriate. 

•  At  least  one  indirect  artifact  and/or  affirmation  exists  to 
confirm  the  implementation. 

•No  substantial  weaknesses  were  noted. 

Largely  Implemented 
(LI) 

•  The  direct  artifact  is  present  and  judged  to  be 
appropriate. 

•  At  least  one  indirect  artifact  and/or  affirmation  exists  to 
confirm  the  implementation. 

•  One  or  more  weaknesses  were  noted. 

Partially  Implemented 

(PD 

•The  direct  artifact  is  absent  or  judged  to  be  inadequate. 
•Artifacts  or  affirmations  suggest  that  some  aspects  of  the 
practice  are  implemented. 

•Weaknesses  have  been  documented. 

Not  Implemented  (NI) 

•Any  situation  not  covered  above 

Continued  on  next  page 
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2.2.2  Characterize  Implementation  of  Model  Practices  (continued) 


Parameters 
and  Limits 
(continued) 


Optional 

Practices 


Implementation 

Guidance 


The  following  table  summarizes  rules  for  aggregating  instantiation-level 
characterizations  to  derive  organizational  unit-level  characterizations. 
Consensus  of  all  members  of  the  appraisal  team  is  necessary  for 
organizational  unit-level  characterizations. 

The  column  labeled  “Condition”  is  the  input  condition — the  practice 
implementation  characterizations  for  the  set  of  sampled  projects.  The  column 
labeled  “Outcome”  is  the  resultant  aggregated  practice  implementation 
characterization  at  the  organizational  unit  level. 


Condition 

Outcome 

Remarks 

All  X  (e.g.,  all  LI) 

X 

All  instantiations  have  the  same  characterization. 

All  (LI  or  FI) 

LI 

All  instantiations  are  characterized  LI  or  higher. 

Any  PI,  No  NI 

LI  or  PI 

Team  judgment  is  applied  to  choose  LI  or  PI  for 
the  organizational  unit. 

Any  NI 

NI,  PI,  or  LI 

Team  judgment  is  applied  to  choose  NI,  PI,  or  LI 
for  the  organizational  unit. 

While  the  initial  characterization  of  practice  implementation  may  be  proposed 
by  a  mini-team  or  some  subset  of  the  team,  the  following  selections  are 
available: 

•  Instantiation-level  characterization  of  practice  implementation  can  be 
reviewed  by  the  entire  team  for  consensus. 

•  Team-wide  review  and  consensus  on  practice  implementation 
characterization  can  be  reserved  for  the  organizational  unit  level. 

•  A  mix  of  the  two  strategies  above,  tailored  to  match  the  learning  curve  of 
the  team  members  or  to  reflect  the  prioritization  of  particular  PAs,  can  be 
used. 


When  the  team  is  ready  to  perform  the  ratings,  these  characterizations  serve  to 
simplify  the  judgments.  The  team  is  then  able  to  focus  on  the  aggregation  of 
weaknesses  observed  to  determine  the  goal  satisfaction  ratings  (explained  in 
process  2.4).  Situations  where  the  project  has  not  yet  reached  the  appropriate 
point  in  time  where  the  practice  would  be  enacted  are  omitted  from  this 
characterization.  The  appraisal-planning  activities  are  expected  to  prevent 
situations  that  severely  limit  the  examples  of  actual  implementation  for  any 
given  practice. 


Continued  on  next  page 


Page  11-174 


CMU/SEI-2002-HB-002 


2.2.2  Characterize  Implementation  of  Model  Practices  (continued) 


Implementation 

Guidance 

(continued) 


The  characterization  of  CMMI  practice  implementation  begins  as  soon  as 
sufficient  data  are  available.  It  is  not  necessary  that  data  for  every  instantiation 
be  available  before  the  implementation  of  any  given  practice  can  be 
characterized  at  the  instantiation  level.  However,  before  the  implementation  of 
a  practice  across  the  organizational  unit  can  be  characterized,  the 
instantiation-level  characterizations  must  be  completed.  Each  instance  of 
practice  enactment  is  characterized  using  the  instantiation-level 
characterization  scheme. 

The  characterization  of  practice  implementation  for  the  organizational  unit  is 
carried  out  using  the  aggregation  rules  summarized  in  the  table  above.  These 
rules  provide  a  basis  for  identifying  the  areas  where  professional  judgment  is 
required,  and  simplify  the  areas  where  the  data  are  unanimous. 
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2.2.2  Characterize  Implementation  of  Model  Practices 


GSS  and  CPM  Implementation  Guidance 


No  additional  guidance. 
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2.2.3  Validate  Practice  Implementation  Gaps 


Activity 

Description 


Required 

Practices 


Verification  activities  lead  to  statements  summarizing  gaps  (weaknesses)  in 
the  implementation  of  model  practices.  Optionally,  statements  reflecting 
exceptional  implementations  of  model  practices  (strengths)  may  also  be 
generated.  These  statements  can  be  generated  at  various  points  in  the  appraisal 
process,  such  as  when 

•  initial  objective  evidence  is  obtained, 

•  implemented  practices  are  compared  to  the  practices  in  the  reference 
model, 

•  the  extent  of  implementation  is  characterized  for  each  project,  or 

•  the  extent  of  implementation  is  characterized  for  the  organizational  unit. 

In  preparation  for  validating  this  information,  the  appraisal  team  generates 
preliminary  findings  that  summarize  the  practice  implementation  gaps.  The 
preliminary  findings  are  written  in  reference  to  a  single  model  practice,  and 
are  abstracted  to  the  level  of  the  organizational  unit.  The  statements  should 
not  reference  a  specific  individual,  project,  or  other  identifiable  organizational 
sub-unit. 

This  is  still  primarily  a  data  collection  activity,  and  the  intent  is  to  validate  the 
appraisal  team’s  understanding  of  the  processes  implemented  within  the 
organizational  unit.  Feedback  from  participants  may  result  in  modifications  to 
the  appraisal  team’s  inventory  of  objective  evidence.  The  results  of  the 
validation  activity  must  be  considered  in  the  formulation  of  final  findings  and 
goal  ratings.  These  latter  activities  cannot  commence  until  after  the  validation 
activity  has  occurred. 


•  Generate  preliminary  findings  summarizing  gaps  in  practice 
implementation  observed  with  the  organizational  unit  relative  to  reference 
model  practices. 

•  Validate  preliminary  findings  with  members  of  the  organizational  unit. 


Continued  on  next  page 
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2.2.3  Validate  Practice  Implementation  Gaps  (continued) 


Parameters 
and  Limits 


Full  appraisal  team  consensus  must  be  reached  on  the  preliminary  findings 
prior  to  providing  them  to  the  organizational  unit  for  validation. 

Preliminary  findings  must  be  corroborated  via  multiple  practice 
implementation  indicator  types  (direct,  indirect,  affirmation).  Areas  where  the 
appraisal  team’s  inventory  of  objective  evidence  is  insufficient  to  satisfy  these 
criteria  may  instead  be  addressed  by  requests  for  additional  information 
needed. 

Preliminary  findings  must  not  refer  to  specific  individuals,  projects,  or 
organizational  sub-units. 

Every  model  practice  characterized  as  either  Not  Implemented  or  Partially 
Implemented,  at  the  organizational  unit  level,  must  have  at  least  one 
preliminary  finding  associated  with  it. 

At  least  one  representative  from  each  project  and  from  any  associated  staff 
functions  must  participate  in  the  set  of  validation  activities. 

Only  appraisal  participants  may  participate  (i.e.,  only  people  who  provided 
data  may  participate  in  validation). 

The  minimum  number  of  validation  sessions  required  is  one,  and  no  more 
than  five  are  recommended,  although  no  maximum  limit  is  specified. 

The  rules  of  confidentiality  and  the  expected  use  of  appraisal  data  must  be 
communicated  to  participants  in  each  validation  activity. 
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2.2.3  Validate  Practice  Implementation  Gaps  (continued) 


Optional  Preliminary  findings  (and  other  appraisal  results)  focused  on  specific  projects, 

Practices  divisions,  or  other  organizational  sub-units  may  be  generated  if  they  are 

reflected  in  the  appraisal  objectives  and  constraints.  This  tailoring  option  also 
requires  that  the  members  of  the  organization  participating  in  the  appraisal  be 
fully  informed  of  the  intended  use  of  the  information  they  provide  to  the 
appraisal  team. 


implementation  Preliminary  findings  are  the  building  blocks  that  lead  to  the  judgment  of  goal 
Guidance  satisfaction,  and  are  the  detailed  information  that  forms  the  basis  for  the  final 

findings.  As  an  intermediate  artifact  of  the  appraisal  process,  preliminary 
findings  are  used  to  ensure  traceability  between  appraisal  inputs  and  appraisal 
outputs. 

Feedback  from  participants  on  the  preliminary  findings  should  be  solicited  by 
the  appraisal  team  and  considered  for  possible  revisions  to  its  inventory  of 
objective  evidence. 

It  is  not  expected  that  preliminary  findings  will  provide  a  detailed  listing  of 
the  implementation  status  of  every  model  practice  in  every  sampled  project. 
Furthermore,  it  is  not  expected  that  the  preliminary  findings  will  identify  the 
status  of  individual  projects  with  regard  to  practice  implementation  or  goal 
achievement.  An  appraisal  sponsor  may  request  these  more  detailed  appraisal 
results.  The  appraisal  team  leader  should  negotiate  for  the  proper  allocation  of 
time  to  accommodate  this  tailoring  option,  and  the  expectation  that  such 
information  will  be  preserved  at  the  end  of  the  appraisal  should  be  made  clear 
to  all  appraisal  participants. 


Continued  on  next  page 
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2.2.3  Validate  Practice  Implementation  Gaps  (continued) 


Implementation 

Guidance 

Preliminary 

Findings 

Presentations 


Implementation 

Guidance 

Focus  Groups 


Implementation 

Guidance 

Survey 

Instrument 


An  interactive  presentation  is  the  most  effect  Mechanism  for  validating  the 
preliminary  findings.  The  members  of  the  or  .ization  w  ho  provided  data  to 
the  appraisal  team  are  typically  brought  together  in  a  conference  room,  and  a 
slide  presentation  is  used  to  review  the  preliminary  findings  in  an  effort  to 
invite  people  to  provide  additional  data,  or  express  their  agreement  with  the 
summary'  statements.  The  audience  is  often  grouped  by  seniority  in  the 
organization,  and  separate  presentations  are  made  for  practitioners,  project 
managers,  and  middle  managers. 

During  the  presentation,  one  or  more  members  of  the  team  review  the 
preliminary  findings  statements  and  provide  the  audience  with  an  opportunity 
to  comment  or  ask  questions.  The  presenter  uses  only  the  words  crafted  by  the 
appraisal  team  and  avoids  elaborating  on  the  findings  using  his  or  her  own 
words.  When  questions  are  asked  about  a  preliminary  finding,  the  team  leader 
provides  any  clarification  needed  to  understand  what  the  statement  means. 
However,  team  members  avoid  the  appearance  that  they  are  justifying  the 
content  of  the  statement. 

The  detailed  data  that  led  to  the  preliminary  findings  must  be  protected,  and 
there  is  no  negotiation  for  wording  or  eliminating  findings.  The  appraisal  team 
must  record  new  data  made  available  to  them  without  commenting  on  how  the 
data  may  be  interpreted  or  how  the  findings  may  need  to  change. 


As  an  alternative  (or  in  addition)  to  the  presentation,  focus  groups  can  be  used 
to  probe  more  deeply  into  specific  areas  of  the  CMMI  model  with  a  targeted 
audience.  This  would  permit  the  team  to  explore  a  particular  area  in  more 
depth  to  help  sharpen  the  appraisal  results,  or  to  raise  the  visibility  of  the 
results  to  people  who  are  most  informed  on  the  topic.  For  example,  a  focus 
group  conducted  with  project  managers  could  be  an  ideal  environment  to 
validate  (and  gather  more  detailed  data  on)  the  topic  of  project  planning  and 
project  monitoring.  In  contrast,  a  focus  group  composed  of  Engineering 
Process  Group  (EPG)  members  may  be  an  ideal  setting  to  validate  findings 
associated  with  the  organization’s  infrastructure  for  process  improvement. 
The  preliminary  findings  that  relate  to  the  group  may  be  distributed  as 
handouts  or  displayed  using  a  projector,  and  the  participants  can  engage  in  a 
free-form  dialogue  with  the  team  and  amongst  themselves.  Notes  taken  by  the 
members  of  the  team  are  treated  as  any  data  collected  during  an  interview 
would  be. 


Finally,  a  survey  instrument  can  be  used  in  addition  (or  as  an  alternative)  to 
either  of  the  techniques  above.  A  carefully  worded  instrument  that  asks 
respondents  to  rate  their  level  of  agreement  with  the  finding  statement,  and 
provides  an  opportunity  for  written  feedback,  can  provide  a  low-cost  and 
timely  source  of  data  for  the  team. 
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2.2.3  Validate  Practice  Implementation  Gaps 


GSS  and  CPM  Implementation  Guidance 

When  using  the  MDD  for  Government  source  selection  and  contract  process  monitoring,  the 
following  should  be  considered: 

Guidance  for  Based  on  the  particular  source  selection  environment  that  the  appraisal  team  is 

GSS  and  CPM  operating  in,  whether  the  appraisal  team  are  members  of  the  SSEB,  and 

whether  “discussions”  have  been  opened,  there  may  be  some  source  selection 
constraints  bearing  on  the  ability  to  validate  preliminary  findings  with  the 
organizational  unit.  If  source  selection  constraints  preclude  validating 
preliminary  findings,  the  appraisal  becomes  an  ARC  Class  B  or  Class  C 
evaluation.  Every  available  option  must  be  exercised  in  the  source  selection 
environment  to  maximize  the  opportunity  for  feedback  and  validation.  Since 
acquisition  organizations  vary  in  the  level  of  interaction  and  feedback 
“allowed”  both  before  and  after  discussions  are  opened,  it  is  important  to 
establish  a  means  to  achieve  this  validation  of  practice  implementation  gaps 
consistent  with  local  source  selection  processes  and  practices.  This  could 
occur  through  formal  communications  via  the  GSS  process;  clarification 
notices,  such  as  Evaluation  Notices,  Requests  for  Clarification,  and  Points  for 
Negotiation;  or  other  approved  feedback  approaches. 

One  approach  to  validating  practice  implementation  gaps,  albeit  still  possibly 
constrained  by  the  GSS  environment,  is  to  conduct  follow-up  focus  group 
interviews,  with  questions  focused  on  the  areas  of  weakness.  This  provides 
another  opportunity  for  the  organization  and  will  increase  confidence  that  the 
preliminary  findings  are  correct,  or  not. 

Note:  Validating  preliminary  findings  is  a  required  activity  of  the  SCAMPI 
method.  If  the  GSS  environment  precludes  this  activity  from  occurring,  then 
an  ARC  Class  B  or  C  appraisal  method  should  be  considered.  See  activity 
1.1.2,  Determine  Appraisal  Constraints. 

For  CPM  in  which  incentives  or  performance  awards  are  involved,  similar 
constraints  and  issues  may  exist  and  should  be  considered  in  the  planning. 
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2.3  Document  Objective  Evidence 


Purpose 

Entry  Criteria 

Inputs 

Activities 

Outputs 

Outcome 

Exit  Criteria 


Create  lasting  records  of  the  information  gathered  by  identifying  and  then 
consolidating  notes,  transforming  the  data  into  records  that  document  practice 
implementation  as  well  as  strengths  and  weaknesses. 


Planning  activities  for  the  appraisal  are  complete,  including  the  selection  and 
preparation  of  the  appraisal  team.  At  least  one  data  collection  activity  has 
been  conducted,  and  appraisal-relevant  data  are  available  to  record. 


Appraisal  data 

•  notes  taken  during  data  collection  activities  (if  applicable) 

•  annotated  worksheets  or  other  work  aids  containing  data  (if  applicable) 

•  strengths  and  weaknesses  documented  from  previous  activities 

•  data  collection  plan 


2.3.1.  Take/Review/Tag  Notes 

2.3.2.  Record  Presence/Absence  of  Objective  Evidence 

2.3.3.  Document  Practice  Implementation  Gaps 

2.3.4.  Review  and  Update  the  Data  Collection  Plan 


•  Updated  appraisal  data 

-  noted  practice  implementation  gaps  (if  any) 
revised  data  collection  plan  (if  applicable) 
annotated  worksheets 

•  Requests  for  additional  data  (interviewees  or  documents) 


Individual  team  members  understand  the  data  collected  thus  far,  and  have 
information  to  guide  any  needed  subsequent  data  collection. 


All  data  from  the  most  recent  data  collection  session  has  been  captured  as  a 
new  baseline  of  practice  implementation  evidence  or  strength  and  weakness 
statements.  The  data-gathering  plans  have  been  updated  to  reflect  additional 
information  needed  and  topics  that  can  be  removed  from  active  investigation. 


Continued  on  next  page 
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2.3  Document  Objective  Evidence  (continued) 


Key  Points 


Tools  and 
Techniques 


Metrics 


Verification  and 
Validation 


Records 


Tailoring 


This  process  has  traditionally  been  the  most  difficult  one  to  manage  during  an 
appraisal.  Members  of  the  team  will  tend  to  vary  a  great  deal  in  their 
productivity  and  style  of  work.  The  team  leader  must  be  very  attentive  to  the 
progress  of  each  team  member,  and  take  effective  corrective  actions  to  ensure 
team  progress. 


Because  of  the  challenging  nature  of  this  activity.  Lead  Appraisers  tend  to 

have  strong  preferences  for  tools  and  techniques  they  have  found  to  be 

successful.  Only  a  high-level  list  of  tools  and  techniques  is  provided  here. 

•  Work  aids  like  wall  charts,  spreadsheet  programs,  and  automated  database 
tools  are  frequently  used  to  help  track  the  status  of  data  collection. 

•  Using  mini-teams,  where  pairs  (or  triplets)  of  team  members  are  assigned 
specific  PAs,  is  a  very  common  practice. 

•  Time  management  is  a  critical  skill  for  this  activity.  Explicitly  reviewing 
the  effort  spent,  in  real  time,  is  a  useful  way  to  focus  the  team. 

•  A  variety  of  techniques  for  structuring  team  notebooks  and  formats  for 
recording  notes  has  been  used. 

•  Team  norms  regarding  techniques  for  managing  debates  and  divergent 
views  are  important,  and  should  be  made  explicit  well  in  advance. 


As  mentioned  above,  tracking  the  effort  expended  during  this  activity  (in  real 
time)  is  a  valuable  technique  to  manage  the  team’s  time.  The  ability  to  quickly 
learn  the  rate  at  which  each  team  member  works  is  a  skill  that  experienced 
Lead  Appraisers  develop  using  effort  and  duration  metrics. 


The  method  rules  for  recording  traceability  and  validating  data  provide  a  lot  in 
the  way  of  verification  and  validation  of  the  appraisal  data.  The  role  of  the 
appraisal  team  leader  in  monitoring  progress  and  the  consensus  decision¬ 
making  process  also  serve  as  important  verification  and  validation  activities. 


All  appraisal  data  are  recorded  with  full  traceability  to  information  sources  as 
well  as  the  model  components  to  which  they  pertain.  The  full  detail  in  this 
traceability  contains  sensitive  information  that  should  not  be  provided  to 
people  outside  of  the  appraisal  team.  The  attribution  of  data  to  individuals  or 
groups  must  never  be  divulged  even  if  some  of  the  detailed  data  are  provided 
to  the  Engineering  Process  Group  at  a  site  for  use  in  process  improvement. 


The  use  of  a  specialized  appraisal  data  management  tool  is  a  common 
tailoring  applied  to  this  activity. 


Continued  on  next  page 
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2.3  Document  Objective  Evidence  (continued) 


The  mechanics  associated  with  the  recording  and  transcription  of  objective 
evidence  are  described  in  this  section.  There  are  many  links  between  these 
mechanics  and  the  data  collection  process,  as  well  as  the  data  verification  and 
validation  process.  It  is  important  to  understand  that  the  data-recording 
process  must  support  these  other  processes,  and  that  the  tools  used  during  an 
appraisal  will  need  to  accommodate  these  linkages.  Typically,  an  integrated 
database  tool  is  used  to  manage  all  appraisal  data  that  results  from  the  analysis 
of  notes  taken  during  data  collection. 

Summary  of  The  most  basic  representation  of  appraisal  data  is  found  in  the  notes  taken  by 

Activities  individual  team  members.  These  notes  are  reviewed  and  are  typically 

“tagged”  or  otherwise  processed  before  their  information  content  is 
transformed  into  other  lasting  representations.  The  presence,  absence,  and/or 
appropriateness  of  objective  evidence  is  then  judged  and  recorded  based  on 
the  data  collected.  The  scheme  by  which  this  set  of  records  is  produced  is  an 
important  implementation  choice  made  by  the  appraisal  team  leader,  and  must 
be  well  understood  by  the  team.  Gaps  in  the  implemented  practices  are  also 
recorded,  in  a  consistent  manner  that  ensures  traceability.  Finally,  the  data 
collection  plan  must  be  reviewed  in  light  of  the  changes  in  the  set  of  data 
available  to  the  team,  and  the  remaining  data  needed  to  support  reliable  rating 
judgments. 

♦ 


Interfaces 
with  Other 
Processes 
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2.3  Document  Objective  Evidence 
GSS  and  CPM  Implementation  Guidance 


No  additional  guidance. 
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2.3.1  Take/Review/Tag  Notes 


Activity 

Description 


Required 

Practices 

Parameters 
and  Limits 


Optional 

Practices 


As  team  members  examine  data  sources,  they  will  document  what  the 
objective  evidence  is  (referencing  documents,  presentations,  instruments,  and 
interviewee  comments),  as  well  as  why  or  how  the  objective  evidence  meets 
the  intent  of  the  model  practice. 

There  may  be  special  cases  where  team  members  elect  to  record  data  directly 
in  the  objective  evidence  tracking  tool.  In  such  cases  the  team  members  may 
choose  not  to  take  notes  (on  paper  or  in  their  notebooks)  that  describe  the 
objective  evidence. 

For  all  interviews  and  presentations,  the  team  members  must  take  notes  that 
capture  the  objective  evidence  before  they  move  to  the  annotation  of  the 
objective  evidence  tracking  tool. 


•  Record  notes  obtained  from  objective  evidence  data-gathering  sessions. 

•  Relate  notes  to  corresponding  practices  in  the  appraisal  reference  model. 


Every  team  member  present  must  take  notes  during  interviews  and 
presentations.  These  notes  must  cover  all  areas  investigated  during  the 
interview,  and  are  not  limited  to  the  PAs  assigned  to  the  individual  team 
member  (i.e.,  everybody  takes  notes  on  everything). 

During  document  reviews  and  the  review  of  instruments,  notes  must  be  taken 
to  preserve  specific  context  or  focused  references,  if  the  rationale  for 
accepting  the  objective  evidence  is  not  self-evident. 

Whenever  notes  are  taken  in  a  data-gathering  session,  individual  team 
members  should  review  their  notes  immediately  after  the  conclusion  of  the 
session.  The  review  will  focus  on  tagging  significant  items  that  relate  to  one 
or  more  model  practice(s).  This  review  and  tagging  process  must  occur  within 
24  hours  of  the  data-gathering  session. 


Tagging  schemes  (that  show  traceability  to  model  practices)  and  techniques 
for  highlighting  phrases  are  determined  by  the  preferences  of  the  note  taker.  A 
variety  of  formats  for  team  member  notebooks  has  been  devised  to  facilitate 
note  taking  and  tracking  raw  data  during  appraisals.  Frequently,  the  questions 
used  during  an  interview  will  be  printed  and  collated  within  a  team  member 
notebook  that  contains  note-taking  forms  and  other  useful  information  like 
interview  schedules  and  document  lists. 

Notes  can  be  recorded  for  items  that  have  significant  positive  or  negative 
impact  upon  the  enactment  of  processes  within  the  organizational  unit,  even  if 
they  are  not  directly  related  to  model  practices.  These  may  ultimately  be 
reflected  in  non-model  findings  reported  to  the  organizational  unit. 


Continued  on  next  page 
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2.3.1  Take/Review/Tag  Notes  (continued) 


Implementation 

Guidance 


Implementation 

Guidance 

Taking  Notes 


Implementation 

Guidance 

Reviewing 

Notes 


Implementation 

Guidance 

Tagging  Notes 


The  raw  notes  taken  during  an  appraisal  are  treated  as  confidential 
information,  and  may  not  be  provided  to  any  person  outside  of  the  appraisal 
team.  Team  members  are  typically  required  to  destroy  their  notes  in  a  secure 
manner  at  the  conclusion  of  the  appraisal.  This  ensures  that  the  attribution  of 
detailed  information  to  individuals  in  the  organization  cannot  lead  to 
inappropriate  consequences  following  the  appraisal. 


Team  members  actively  take  notes  during  all  data-gathering  sessions.  The 
purpose  is  to  record,  verbatim,  what  the  information  source  reveals  about  the 
implementation  of  practices  in  the  project  or  organization.  Note-taking  is  done 
for  all  types  of  objective  evidence: 

•  The  analysis  of  instruments  yields  information  and  references  regarding 
the  implementation  of  practices,  ideally  with  traceability  to  the  model. 

•  While  reviewing  documents  it  is  often  important  to  note  a  specific  phrase 
or  reference  and  to  record  the  document  name  and  page  number. 

•  When  receiving  presentations,  phrases  or  references  provided  as 
elaboration  on  presentation  material  are  captured  in  notes. 

•  Interviews  are  the  most  intensive  activity  with  regard  to  note  taking.  The 
purpose  is  to  record  what  the  interviewees  said;  not  what  the  team 
member  believes  they  meant. 


The  context  in  which  the  data  are  provided — be  it  during  an  interview, 
presentation,  or  in  a  document — bears  on  the  proper  interpretation  of  the  facts. 
For  example,  notes  taken  during  an  interview  are  based  on  a  give  and  take 
between  the  interviewer  and  the  interviewee.  The  threads  of  discussion  often 
provide  a  context  that  may  not  be  reflected  in  a  single  excerpt  from  the  middle 
of  the  interview.  Note-takers  should  review  their  work  to  ensure  that  such 
contextual  information  can  be  preserved  at  least  in  their  recollection,  and 
preferably  through  the  annotation  of  the  notes. 


As  notes  are  reviewed,  team  members  often  use  highlighter  pens  or  annotation 
schemes  to  identify  the  most  salient  excerpts.  The  PA  and/or  practice  to  which 
the  information  applies  may  be  written  in  colored  ink  over  the  raw  notes.  All 
notes  should  identify  the  data-gathering  session,  and  the  pages  should  be 
numbered  to  preserve  the  sequence  of  information.  For  notes  taken  during 
interviews,  it  is  often  very  useful  to  draw  a  seating  chart  to  show  where  each 
person  was  sitting  during  the  interview.  Scripts  prepared  in  advance  of 
scheduled  interviews  may  already  be  tagged,  and  can  help  relate  responses  to 
appropriate  sections  of  the  reference  model.  Some  interviewee  responses  may 
deal  with  model  practices  other  than  those  targeted  by  a  given  question,  which 
would  still  necessitate  some  additional  tagging. 
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2.3.1  Take/Review/Tag  Notes 


GSS  and  CPM  Implementation  Guidance 

When  using  the  MDD  for  Government  source  selection  and  contract  process  monitoring,  the 
following  should  be  considered: 

Guidance  for  Team  members  are  typically  required  to  destroy  notes  at  the  conclusion  of  the 

GSS  and  CPM  appraisal.  In  both  source  selection  and  contract  process  monitoring,  the 

resultant  worksheets  (including  practice  implementation  gaps  and 
characterization)  and  presentations  may  be  retained  until  contract  award  or 
incentives  and/or  performance  award  decisions  are  final:  Information 
presented  outside  of  the  team  must  be  free  of  references  to  specific  individuals 
or  projects. 
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2.3.2  Record  Presence/Absence  of  Objective  Evidence 


Activity 

Description 


Required 

Practices 

Parameters 
and  Limits 


The  presence  or  absence  of  appropriate  objective  evidence  for  each  model 
practice  in  the  scope  of  the  appraisal  is  determined  based  on  information 
obtained  from  data-gathering  sessions.  Annotations  are  recorded  indicating 
the  source,  relevance,  and  coverage  of  objective  evidence  collected.  In 
situations  where  just  referencing  the  data  source  would  not  make  it  obvious 
why  the  objective  evidence  is  appropriate,  a  comment  can  be  added  to  the 
annotation.  For  example,  when  an  alternative  to  the  typical  work  breakdown 
structure  is  used,  it  may  be  necessary  to  document  why  that  alternative  meets 
the  intent  of  the  model  practice.  Adding  comments  to  the  annotations  can  help 
to  avoid  rehashing  the  rationale  for  accepting  the  objective  evidence  multiple 
times  during  team  discussions. 


Record  the  presence  or  absence  of  appropriate  objective  evidence  collected  for 
each  reference  model  practice. 


The  inventory  of  objective  evidence  (be  it  in  electronic  or  paper  form)  is 
updated  to  reflect  what  the  data  imply  about  the  implementation  of  particular 
practices.  For  every  practice  within  the  model  scope  of  the  appraisal, 
annotations  indicating  the  presence  or  absence  of  objective  evidence  will  be 
made  throughout  the  appraisal  conduct.  The  annotation  scheme  used  must 
ensure  that  the  record  reveals  the  following  information: 

•  the  project  or  organizational  unit  to  which  the  data  apply 

•  the  specific  or  generic  practice  to  which  the  data  apply 

•  the  type  of  objective  evidence  being  recorded  (i.e.,  direct,  indirect,  or 
affirmation) 

•  whether  the  data  imply  the  presence  or  absence  of  the  objective  evidence 

•  whether  the  data  suggest  that  the  objective  evidence  is  appropriate 

•  comments  about  the  appropriateness  of  the  evidence  (if  needed) 

•  whether  or  not  additional  information  is  needed  before  the  team  can 
characterize  the  extent  to  which  the  practice  is  implemented 

•  a  description  of  what  the  evidence  is,  if  such  a  description  was  not 
provided  by  the  organization  in  advance 


Continued  on  next  page 
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2.3.2  Record  Presence/Absence  of  Objective  Evidence  (continued) 


Optional 

Practices 


Implementation 

Guidance 


Page  11-196 


Following  each  verification  session  where  the  presence  or  absence  of 
objective  evidence  is  recorded,  the  team  reviews  the  judgments  about  each 
new  piece  of  objective  evidence.  This  may  be  useful  in  establishing  a  common 
understanding  of  the  expectations  for  objective  evidence,  especially  early  in 
the  appraisal. 


This  activity  represents  the  mechanical  aspects  of  processing  appraisal  data, 
and  is  strongly  tied  to  the  activities  described  in  process  2.2,  Verify  and 
Validate  Objective  Evidence.  The  emphasis  of  this  activity  description  is  on 
the  steps  needed  to  update  the  inventory  of  objective  evidence  arid  rriaintain 
traceability  to  data  sources.  The  emphasis  of  the  activity  description  in  Verify 
and  Validate  Objective  Evidence  is  on  the  interpretation  of  data  collected  and 
the  sufficiency  of  objective  evidence  relative  to  the  appraisal  reference  model. 

Team  members  typically  record  the  presence  or  absence  of  appropriate 
objective  evidence  into  tools  such  as  tracking  tables  or  data  consolidation 
worksheets.  Prior  to  the  assignment  of  goal  ratings,  the  entire  team  reviews 
the  status  of  the  objective  evidence  as  reflected  in  the  annotations  made  by 
each  team  member. 

The  data  gathered  during  every  data  collection  session  should  be  related  to  the 
practices  in  use  in  a  project  or  across  the  organization.  In  recording  the 
presence  or  absence  of  objective  evidence,  the  intent  is  to  quickly  inventory 
the  composite  of  factual  information.  Elaboration  about  what  the  data  mean  or 
how  they  relate  to  other  important  issues  is  captured  either  in  notes  or  in  the 
descriptions  of  practice  implementation  gaps  crafted  by  team  members. 


♦ 
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2.3.3  Document  Practice  Implementation  Gaps 


Activity 

Description 


Required 

Practices 

Parameters 
and  Limits 


The  primary  intent  of  this  activity  is  to  derive,  from  the  objective  evidence 
gathered,  summary  prose  statements  that  describe  the  gap  between  what  the 
objective  evidence  shows  and  what  the  team  was  looking  for  to  support  a 
claim  that  the  model  practice  was  implemented.  The  statements  explain  why 
the  practice  is  not  considered  to  be  Fully  Implemented.  Statements  of  practice 
implementation  gaps  will  be  validated  with  the  organizational  unit  at  a  later 
time. 

Strengths  are  not  recorded  pro  forma  when  practices  are  found  to  be  Fully 
Implemented.  Where  practices  represent  exemplary  implementations  of  the 
model  practices,  the  appraisal  team  will  highlight  these  as  part  of  the  appraisal 
output.  However,  the  primary  focus  of  this  benchmarking  method  is  to  help 
the  organization  verify  the  implementation  of  the  model  and  identify  areas 
where  work  is  needed. 


Describe  in  writing  gaps  in  the  organizational  unit’s  implemented  processes 
relative  to  reference  model  practices. 


For  any  practice  that  is  characterized  as  something  other  than  Fully 
Implemented,  there  must  be  a  prose  statement  explaining  the  gap  between 
what  the  organization  does  and  what  the  model  expects. 

Regardless  of  the  medium  used,  statements  describing  practice 
implementation  gaps  should  be  annotated  with  the  following  identifying 
information: 

•  the  model  component  to  which  the  statement  relates  (i.e.,  PA,  goal,  and 
practice) 

•  the  data  collection  session(s)  in  which  the  information  was  uncovered 

•  the  process  instantiation(s)  to  which  the  statement  applies 

Prose  statements  of  practice  implementation  gaps  presented  to  the 
organizational  unit  in  the  form  of  preliminary  findings  for  validation  must  be 
free  of  references  to  specific  individuals  or  projects. 


Continued  on  next  page 
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2.3.3  Document  Practice  Implementation  Gaps  (continued) 


Optional  Document  strengths  in  the  implementation  of  model  practices  when  the  team 

Practices  discovers  exemplary  implementations. 

Label  implementation  gaps  as  “opportunities  for  improvement”  to  avoid  the 
potentially  negative  connotations  of  labeling  them  as  weaknesses. 

Document  any  significant  issues  impeding  performance  in  the  organization 
that  do  not  necessarily  map  to  the  CMMI  model.  This  must  be  done 
cautiously,  and  the  number  of  these  issues  should  not  be  larger  than  the 
number  of  model-related  issues  reported  by  the  team. 


Implementation  The  database  used  to  record  the  inventory  of  objective  evidence  may 
Guidance  incorporate  functionality  to  record  practice  implementation  gaps  and 

strengths,  or  a  separate  location  or  tool  may  be  used  if  desired.  Gaps  in 
practice  implementation  should  be  recorded  at  the  level  of  a  particular 
instance  of  a  model  practice.  These  precursors  to  preliminary  findings  are 
more  detailed  and  pointed,  while  all  information  presented  outside  of  the  team 
will  be  aggregated  to  the  goal  and  organizational  unit  level  of  abstraction. 

Strengths  are  only  documented  if  the  implementation  of  a  practice  is 
exceptional,  and  reflects  a  strong  asset  in  the  process  in  use.  An  adequate 
implementation  of  a  model  practice  is  not  necessarily  a  strength.  Team 
members  should  use  their  collective  experience  and  judgment  to  determine 
whether  or  not  they  have  uncovered  an  exemplary  practice  (above  and  beyond 
the  capability  described  in  the  model)  to  highlight  in  the  appraisal  output. 

Gaps  in  practice  implementation  are  documented  if  the  objective  evidence 
indicates  a  missing  component  in  the  process  or  an  inappropriate  practice,  in 
light  of  the  value  the  practice  is  expected  to  add  to  the  achievement  of  the 
goal.  That  is,  practices  that  fail  to  help  the  organization  meet  the  CMMI  goal 
to  which  they  relate  should  have  a  gap  documented  that  explains  why  the  goal 
is  not  met. 


♦ 
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2.3.3  Document  Practice  Implementation  Gaps 
GSS  and  CPM  Implementation  Guidance 

When  using  the  MDD  for  Government  source  selection  and  contract  process  monitoring,  the 
following  should  be  considered: 

Guidance  for  During  this  activity,  the  decisions  made  and  documented  in  activity  1.1.4, 
GSS  and  CPM  Determine  Outputs,  become  executable.  The  decision  regarding  whether  to 
document  practice-level  ratings  in  addition  to  the  SCAMPI  goal  ratings  and 
findings  of  strengths  and  weaknesses  will  determine  the  level  of  information 
available  to  the  SSEB  (for  GSS)  or  to  the  awards  official  (for  CPM). 
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2.3.4  Review  and  Update  the  Data  Collection  Plan 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Optional 

Practices 


Implementation 

Guidance 


This  activity  is  used  to  continuously  monitor  the  state  of  available  objective 
evidence  and  to  select  the  next  tactic  in  the  pursuit  of  obtaining  full  coverage 
of  the  model  and  the  organizational  unit. 


•  Review  the  inventory  of  objective  evidence  collected  and  the  data 
collection  plan  to  determine  what  additional  objective  evidence  is  still 
needed  for  sufficient  coverage  of  the  model  scope. 

•  Revise  the  data  collection  plan  to  obtain  additional  objective  evidence  for 
instances  where  insufficient  data  are  available  to  judge  the 
implementation  of  reference  model  practices. 

•  Identify  priorities  for  the  upcoming  data  collection  events,  and  reevaluate 
the  feasibility  of  the  schedule  in  light  of  the  current  state  of  the  objective 
evidence. 


This  activity  must  be  enacted  at  least  once  a  day,  and  a  consolidated  summary 
of  the  appraisal  data  collection  status  must  be  available  to  the  team  at  the  start 
of  each  day  during  which  data  collection  events  are  planned. 


In  addition  to  the  daily  status  mentioned  above,  more  frequent  status  checks 
may  be  conducted.  These  interim  status  checks  are  not  aggregated  across  the 
team,  for  a  team-wide  view  of  status,  unless  the  appraisal  team  leader  finds 
that  beneficial. 


The  data  collection  status  summarizes  the  differences  between  the  objective 
evidence  on  hand  and  the  evidence  needed  to  support  the  creation  of  appraisal 
outputs  (e.g.,  ratings).  Annotations  regarding  the  presence  (and 
appropriateness)  of  objective  evidence  allow  the  team  to  inventory  the  state  of 
the  “knowledge  base.”  This  status  then  drives  requirements  for  the  collection 
of  more  data,  which  must  be  met  by  the  data  collection  plan.  The  annotation 
of  the  inventory  of  objective  evidence  is  described  in  process  2.2,  Verify  and 
Validate  Objective  Evidence. 

The  plan  for  future  data  collection  should  be  revisited  and  updated  as 
necessary.  There  may  be  several  situations  in  which  additional  data  are 
required  for  the  team  to  sufficiently  characterize  the  implementation  of 
reference  model  practices.  The  following  are  examples  of  such  situations: 

•  The  process  of  reconciling  new  data  with  old  may  identify  conflicts  or 
ambiguities  in  the  data  that  require  clarification. 

•  The  search  for  objective  evidence  may  lead  to  the  discovery  of  one  or 
more  previously  undocumented  practice(s)  in  the  organization. 

•  Attempts  to  confirm  the  use  of  a  particular  practice  or  tool  in  a  project 
may  have  been  unsuccessful. 


Continued  on  next  page 


CMU/SEI-2002-HB-002 


Page  11-203 


2.3.4  Review  and  Update  the  Data  Collection  Plan  (continued) 


Implementation 

Guidance 

(continued) 


Prioritizing  data  needs  and  allocating  data  collection  effort  to  particular  data 
collection  events  are  ongoing  activities  that  the  appraisal  team  leader  is 
responsible  for  overseeing.  The  data  collection  status  summary  may  be 
performed  by  the  appraisal  team  leader  and  reported  to  the  team  members,  or 
the  appraisal  team  leader  may  elect  to  have  each  mini-team  perform  this 
activity  for  the  PAs  it  is  assigned. 

Specific  information  needed  to  resolve  ambiguities  or  conflicts  in  the  existing 
data  should  be  documented  for  follow-up  by  one  or  more  members  of  the 
team.  For  detailed  data  items  that  have  a  limited  scope  of  impact,  the  notes  of 
individual  team  members  may  be  adequate  to  document  the  data  needed.  For 
example,  whether  or  not  a  particular  person  is  involved  in  a  meeting,  or 
reviews  a  given  document,  can  be  confirmed  by  a  simple  question  asked 
during  an  on-call  interview.  Therefore,  a  note  made  by  an  individual  team 
member  to  make  sure  the  question  is  asked  may  suffice.  In  contrast,  if 
conflicting  information  is  uncovered  about  whether  or  not  a  given  event 
occurred,  like  a  meeting,  more  visibility  of  this  conflict  may  be  needed  among 
the  team  members  to  understand  why  the  information  collected  thus  far  is  not 
internally  consistent.  In  such  a  case,  the  person(s)  responsible  for  the  PA 
where  that  practice  resides  may  need  to  alert  the  team  to  the  conflicting  data 
and  facilitate  a  team  discussion  to  seek  clarity,  as  well  as  additional  data.  This 
may  lead  to  the  crafting  of  a  specific  interview  question,  which  is  used  in  a 
standard  interview. 

The  data  collection  plan  and  inventory  of  objective  evidence  provide  a  means 
for  the  appraisal  team  to  continuously  monitor  progress  toward  sufficient 
coverage  of  reference  model  practices  in  preparation  for  rating.  Estimates  of 
the  additional  data  collection  effort  should  be  regularly  reviewed.  If  the 
feasibility  of  the  appraisal  schedule  is  called  into  question,  a  replanning  effort 
may  be  necessary  (as  described  in  activity  1.5.3,  Replan  Data  Collection). 


♦ 
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2.3.4  Review  and  Update  the  Data  Collection  Plan 


GSS  and  CPM  Implementation  Guidance 

When  using  the  MDD  for  Government  source  selection  and  contract  process  monitoring,  the 
following  should  be  considered: 

Guidance  for  In  this  activity,  the  data  collection  plan  (from  process  1.5,  Prepare  for 

GSS  Collection  of  Objective  Evidence)  is  reviewed  at  least  once  per  day.  This 

allows  appraisal  team  members  to  gauge  their  progress  in  gathering  the 
necessary  evidence  to  develop  findings  and  apply  ratings  judgments.  Because 
the  SCAMPI  team  must  work  within  the  overall  time  frame  for  source 
selection,  they  may  not  have  the  option  of  rescheduling  or  extending  the  on 
site  even  if  objective  evidence  for  each  process  instance  is  not  adequate  and 
additional  time  is  needed.  Despite  the  results  of  the  readiness  review  (activity 
1.4.4,  Inventory  Objective  Evidence)  that  indicated  a  verification-based 
appraisal,  the  appraisal  team  may  have  to  work  in  discovery  mode,  which  may 
lead  to  long  hours  or  the  inability  to  achieve  appraisal  objectives  within 
defined  estimates  and  constraints.  An  alternative  approach  is  to  conduct  the 
appraisal  in  advance  of  the  source  selection  (prior  to  RFP  release).  Once 
offerors  make  their  intentions  to  bid  known,  the  Government  can  ask  for  an 
appraisal.  This  allows  more  time  for  possible  rescheduling  or  extension  of  the 
on  site. 

Guidance  for  Time  on  site,  while  typically  constrained,  is  more  easily  negotiated  with  the 
CPM  sponsor  and  the  appraised  organization. 
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2.4  Generate  Appraisal  Results 


Purpose 

Entry  Criteria 

Inputs 

Activities 

Outputs 

Outcome 

Exit  Criteria 


Rate  goal  satisfaction  based  upon  the  extent  of  practice  implementation 
throughout  the  organizational  unit.  The  extent  of  practice  implementation  is 
judged  based  on  validated  data  (e.g.,  direct,  indirect,  and  affirmation  objective 
evidence)  collected  from  the  entire  representative  sample  of  the  organizational 
unit.  The  rating  of  capability  levels  and/or  maturity  levels  is  driven  by  the  goal 
satisfaction  ratings. 


The  set  of  validated  preliminary  findings,  statements  of  practice 
implementation  gaps,  and/or  tabulations  of  validated  objective  evidence  of 
practice  implementation  on  which  they  are  based  are  available.  Team 
members  are  confident  that  they  have  obtained  all  the  pertinent  data  they  will 
need  to  make  rating  judgments.  The  data  obtained  completely  covers  the 
practices  within  the  defined  CMMI  model  scope  and  the  entire  representative 
sample  selected  for  the  organizational  unit. 


Appraisal  data 

•  validated  preliminary  findings 

•  tabulations  of  objective  evidence  of  practice  implementation 

•  annotated  worksheets,  checklists,  working  notes 


2.4. 1  Derive  Findings  and  Rate  Goals 

2.4.2a  Determine  Process  Area  Capability  Level 
2.4.3a  Determine  Capability  Profile 
2.4.2b  Determine  Satisfaction  of  Process  Areas 
2.4.3b  Determine  Maturity  Level 
2.4.4  Document  Appraisal  Results 


•  Final  findings 

•  Recorded  rating  decisions 


A  formal  rating  decision  for  each  reference  model  component  that  was 
planned  to  be  rated,  and  for  which  the  team  obtained  complete  or  sufficient 
data 


Ratings  against  all  components  per  the  plan  have  been  made  and  recorded. 


Continued  on  next  page 
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2.4  Generate  Appraisal  Results  (continued) 


Key  Points 

Tools  and 
Techniques 

Metrics 

Verification  and 
Validation 

Records 

Tailoring 


The  judgment  of  goal  satisfaction  is  based  upon  and  traceable  to  the  extent  of 
the  implementation  of  practices  associated  with  that  goal  (or  alternative 
practices  contributing  equivalently  to  goal  satisfaction). 

Success  in  this  activity  is  driven  by  team  members’  ability  to  limit  their  focus 
to  the  data  that  support  the  judgments,  and  to  avoid  issues  that  threaten  their 
ability  to  be  objective.  This  activity  can  create  a  great  deal  of  stress  for  team 
members  under  pressure  to  help  their  organization  “do  well”;  the  team  leader 
must  skillfully  facilitate  this  activity  when  external  pressures  exist. 


There  is  a  significant  amount  of  data  to  review  in  making  each  round  of 
judgments.  Rating  worksheets  and  automated  support  tools  facilitate  the 
team’s  decision-making  process  by  presenting  necessary  data  in  a  concise, 
well-organized  manner.  When  controversial  issues  are  encountered,  the  team 
leader  must  actively  facilitate  to  ensure  that  the  team  remains  focused  on  the 
pertinent  issues.  Strategic  rest  breaks,  and  sequencing  and  pacing  critical 
discussions,  are  often  keys  to  success. 


•  Planned  versus  actual  effort  for  each  component  rated 

•  Number  of  model  components  rated  satisfied  or  unsatisfied 


The  team  leader  verifies  that  the  rating  process  was  performed  in  accordance 
with  the  method  rules  and  the  rating  baseline  selected  and  documented  in  the 
appraisal  plan.  Work  aids  used  to  record  the  team  judgments  help  ensure 
traceability  to  the  basis  for  the  rating  judgments. 


A  worksheet  or  other  work  aid  may  be  used  to  make  a  record  of  the  rating 
decisions.  A  Process  Area  Profile  is  often  an  effective  means  of  recording  and 
communicating  these  results. 


The  method  provides  tailoring  options  for  rating  additional  model 
components.  The  minimum  requirement  is  to  rate  the  specific  and  generic 
goals  associated  with  each  PA  in  the  scope  of  the  appraisal.  In  addition,  the 
sponsor  may  request  that  maturity  level  and/or  capability  level  ratings  be 
performed  and  reported.  Through  negotiation  between  the  appraisal  team 
leader  and  the  appraisal  sponsor,  a  decision  to  rate  individual  practices  can 
also  be  made. 


Continued  on  next  page 
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2.4  Generate  Appraisal  Results  (continued) 


Interfaces 
with  Other 
Processes 


Summary  of 
Activities 


The  rating  judgments  made  by  the  appraisal  team  members  are  dependent  on 
the  quality  of  the  data  available  to  them,  as  well  as  their  ability  to  reliably 
judge  the  implementation  and  institutionalization  of  practices  in  the 
organization  that  relate  to  the  CMMI  model.  All  the  processes  previously 
described  contribute  to  the  team’s  ability  to  effectively  execute  this  process. 
The  Analyze  Requirements  process  establishes  the  rating  baseline,  the 
organizational  unit  to  which  ratings  will  apply,  and  the  purpose  for  which  the 
ratings  will  be  used.  The  Develop  Appraisal  Plan  process,  in  conjunction  with 
the  Obtain  and  Analyze  Initial  Objective  Evidence  and  Prepare  for  Collection 
of  Objective  Evidence  processes,  determine  the  sample  of  the  organizational 
unit  for  which  data  will  be  collected  and  from  which  the  ratings  will  be 
determined.  The  Select  and  Prepare  Team  process  ensures  that  the  team  has 
sufficient  knowledge  and  skills  to  interpret  the  data  and  arrive  at  sound  rating 
judgments.  The  Examine  Objective  Evidence  and  Document  Objective 
Evidence  processes  provide  the  basic  information  that  is  needed  to  support 
judgments  in  a  form  that  facilitates  making  the  judgments.  The  Verify  and 
Validate  Objective  Evidence  process  characterizes  the  extent  to  which  the 
organization  implements  practices  in  the  model  (or  acceptable  alternatives) 
and  Validates  findings  describing  any  weaknesses  associated  with  the  practice 
implementations.  Upon  the  successful  execution  of  these  processes,  the  team 
is  ready  to  rate  the  satisfaction  of  goals  dependent  on  those  practices. 


The  required  and  fundamental  rating  activity  involves  making  team  judgments 
about  goal  satisfaction  for  each  and  every  specific  and  generic  goal  within  the 
model  scope  defined  in  the  rating  baseline.  Once  goal  satisfaction  has  been 
determined,  optional  rating  activities  can  be  performed  in  accordance  with  the 
defined  rating  baseline  and  the  selected  model  representation(s)  (continuous, 
staged,  or  both).  The  first  optional  activity  focuses  on  rolling  up  goal 
satisfaction  to  PA  ratings.  The  team  determines  a  PA  capability  level  rating  (0 
through  5)  for  each  PA  in  the  continuous  representation  that  is  within  the 
appraisal  scope,  and/or  the  team  determines  a  Satisfied/Unsatisfied  rating  for 
each  PA  in  the  staged  representation  that  is  within  the  appraisal  scope.  The 
second  optional  activity  continues  the  rating  roll  up  to  cover  all  PAs  within 
the  selected  model  scope.  In  the  case  of  the  continuous  representation  the 
team  creates  a  profile  showing  the  capability  levels  for  all  PAs  considered. 
The  profile  can  then  be  used  to  compute  a  maturity  level  through  the 
equivalent  staging  described  in  the  model.  In  the  case  of  the  staged 
representation  the  team  assigns  a  maturity  level  rating  (1  through  5) 
corresponding  to  the  highest  level  in  the  model  for  which  all  applicable  PAs 
have  been  rated  as  satisfied.  The  optional  activities  described  in  2.4.2a  and 
2.4.3a  cover  the  continuous  representation;  those  in  2.4.2b  and  2.4.3b  cover 
the  staged  representation.  As  indicated,  these  options  are  not  mutually 
exclusive. 
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2. 


Generate  Appraisal  Results 


GSS  and  CPM  Implementation  Guidance 


No  additional  guidance. 
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2.4.1  Derive  Findings  and  Rate  Goals 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


The  judgments  made  about  goal  satisfaction  are  driven  by  the  findings  that 
were  documented  by  the  appraisal  team  and  validated  by  appraisal 
participants.  The  preliminary  findings  focus  on  gaps  in  the  implementation  of 
practices.  When  performing  goal  ratings,  the  team  must  judge  whether  or  not 
these  gaps  in  the  implementation  of  practices  (in  aggregate)  threaten  the 
organization’s  ability  to  satisfy  the  goals  associated  with  the  practices. 


•  Derive  final  findings  using  preliminary  findings  statements,  feedback 
from  validation  activities,  and  any  additional  objective  evidence  collected 
as  a  result  of  the  validation  activity. 

•  Rate  each  specific  goal  and  generic  goal  within  the  reference  model  scope 
of  the  appraisal,  based  on  the  practice  implementation  characterizations  at 
the  organizational  unit  level,  as  well  as  the  aggregation  of  weaknesses 
associated  with  that  goal. 

•  Obtain  appraisal  team  consensus  on  the  practice  implementation 
characterizations,  findings  statements,  and  ratings  generated  for  the 
organizational  unit  level. 


When  deriving  final  findings,  the  aim  is  to  create  goal-level  statements  that 
summarize  the  gaps  in  practice  implementation.  These  statements  must  be 
abstracted  to  the  level  of  the  organizational  unit,  and  cannot  focus  on 
individual  projects  (unless  the  tailoring  option  for  project-specific  findings  has 
been  agreed  upon  during  planning). 

If  there  are  no  findings  that  document  the  weaknesses  associated  with  a  goal, 
the  goal  must  be  satisfied. 

The  goal  is  rated  Satisfied  if 

•  all  associated  practices  are  characterized  at  the  organizational  unit  level  as 
either  Largely  Implemented  or  Fully  Implemented,  and 

•  the  aggregation  of  weaknesses  associated  with  the  goal  does  not  have  a 
significant  negative  impact  on  goal  achievement. 

For  a  goal  to  be  rated  as  Unsatisfied,  the  team  must  be  able  to  describe  how 
the  set  of  weaknesses  (or  single  weakness)  led  to  this  rating. 


Continued  on  next  page 
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2.4.1  Derive  Findings  and  Rate  Goals  (continued) 


Optional 

Practices 


Implementation 

Guidance 


Page  11-214 


Findings  statements  and  satisfaction  ratings  may  be  specified  at  the  level  of 
individual  practices  if  the  appraisal  sponsor  specifically  requests  this  tailoring 
option.  These  practice-level  ratings  must  be  based  on  the  extent  to  which  the 
implemented  practice  (or  the  absence  of  implementation)  supports  the 
achievement  of  the  related  goal.  The  use  of  informative  material  to  form  a 
checklist  is  explicitly  discouraged.  A  rating  algorithm  for  practices  that  does 
not  have  a  demonstrable  link  to  PA  goals  would  depart  from  the  intended  use 
of  CMMI  components. 


Any  endeavor  that  results  in  producing  a  score,  grade,  or  rating  is  by 
definition  an  area  of  sensitivity  to  those  affected  by  the  outcome.  An  objective 
and  clear-cut  basis  for  assigning  a  rating  lessens  this  sensitivity  and  results  in 
a  more  consistent  basis  of  comparison  among  the  organizational  units  and 
goals  rated.  Judgments  made  prior  to  and  during  the  rating  process  should  be 
based  on  observable  facts  and  should  be  made  at  the  lowest  level  of 
abstraction  that  makes  sense.  In  the  case  of  CMMI,  the  lowest  level  of 
abstraction  is  characterizing  the  extent  of  practice  implementation  for  each 
process  instantiation  within  the  representative  sample.  Characterizations  made 
at  the  instantiation  level  are  aggregated  into  a  characterization  of  the  extent  of 
practice  implementation  throughout  the  organization,  as  described  earlier  in 
process  2.2,  Verify  and  Validate  Objective  Evidence.  The  judgment  of  goal 
satisfaction  is  then  based  upon,  and  directly  traceable  to,  the  extent  of 
implementation  of  practices  associated  with  that  goal  (or  alternative  practices 
contributing  equivalently  to  goal  satisfaction). 

Findings  should  be  phrased  in  terms  that  best  facilitate  decision  making  by  the 
appraisal  sponsor  and  taking  action  upon  the  appraisal  results. 


♦ 
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2.4.1  Derive  Findings  and  Rate  Goals 
GSS  and  CPM  Implementation  Guidance 

No  additional  guidance. 


2.4.2a  Determine  Process  Area  Capability  Level 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


When  using  the  continuous  representation  of  a  CMMI  model,  the  team  may 
make  rating  judgments  about  each  PA  (and  associated  capability  level)  within 
the  scope  of  the  appraisal.  Assigning  capability  level  ratings  is  an  optional 
activity,  selected  at  the  discretion  of  the  appraisal  sponsor  and  documented  in 
the  appraisal  input. 


Rate  the  capability  levels  for  each  PA  within  the  scope  of  the  appraisal,  based 
upon  the  highest  level  and  all  levels  below  for  which  its  specific  goals  and  the 
generic  goals  within  the  appraisal  scope  have  been  satisfied  (if  this  rating 
option  was  selected  during  planning). 


The  table  below  provides  the  criteria  for  deriving  the  capability  level  rating 
for  each  PA. 


Capability 

Level 

Engineering  Process  Areas 

Other  Process  Areas 

0 

Default  Rating 

Default  Rating 

1 

Generic  goal  for  capability  level  1  is 
rated  Satisfied. 

All  specific  goals  are  rated  Satisfied  - 
including  base  practices  only. 

Generic  goal  for  capability 
level  1  is  rated  Satisfied. 

All  specific  goals  are  rated 
Satisfied. 

2 

Generic  goals  for  capability  levels  1 
and  2  are  rated  Satisfied. 

All  specific  goals  are  rated  Satisfied  - 
including  specific  practices  at 
capability  levels  1  and  2. 

Generic  goals  for  capability 
levels  1  and  2  are  rated 
Satisfied. 

All  specific  goals  are  rated 
Satisfied. 

3 

Generic  goals  for  capability  levels  1, 2, 
and  3  are  rated  Satisfied. 

All  specific  goals  are  rated  Satisfied  - 
including  specific  practices  at 
capability  levels  1, 2,  and  3. 

Generic  goals  for  capability 
levels  1,2,  and  3  are  rated 
Satisfied. 

All  specific  goals  are  rated 
Satisfied. 

4 

Generic  goals  for  capability  levels  1, 2, 
3,  and  4  are  rated  Satisfied. 

All  specific  goals  are  rated  Satisfied  - 
including  specific  practices  at 
capability  levels  1, 2,  and  3. 

Generic  goals  for  capability 
levels  1, 2,  3,  and  4  are  rated 
Satisfied. 

All  specific  goals  are  rated 
Satisfied. 

5 

Generic  goals  for  capability  levels  1,  2, 
3, 4,  and  5  are  rated  Satisfied. 

All  specific  goals  are  rated  Satisfied  - 
including  specific  practices  at 
capability  levels  1, 2,  and  3. 

Generic  goals  for  capability 
levels  1, 2, 3, 4,  and  5  are 
rated  Satisfied. 

All  specific  goals  are  rated 
Satisfied. 

Continued  on  next  page 


CMU/SEI-2002-HB-002 


Page  11-217 


2.4.2a  Determine  Process  Area  Capability  Level  (continued) 


Optional  The  rating  of  PA  capability  levels  may  be  carried  out  top  down  or  bottom  up, 

Practices  as  described  below. 

The  bottom  up  approach  uses  the  following  sequence: 

•  Judge  whether  or  not  the  PA  can  be  considered  to  be  at  capability  level  1, 
based  on  the  satisfaction  of  specific  and  generic  goals.  In  this  case,  only 
the  base  practices  would  be  considered  in  rating  goals. 

•  Judge  whether  or  not  the  PA  can  be  considered  to  be  at  capability  level  2, 
based  on  the  satisfaction  of  specific  and  generic  goals.  In  this  case,  the 
advanced  practices  for  capability  level  2  must  be  considered  in  rating  the 
goals  of  the  Engineering  PAs. 

•  Proceed  incrementally  until  the  team  reaches  a  point  at  which  the  goals 
cannot  be  rated  as  satisfied. 

The  top  down  approach  uses  the  following  sequence: 

•  Begin  at  the  highest  desired  capability  level  (which  was  determined 
during  appraisal  planning)  and  judge  whether  or  not  the  PA  can  be 
considered  to  be  operating  at  that  capability  level. 

•  If  the  PA  is  not  at  the  highest  desired  capability  level,  consider  whether  or 
not  it  can  be  judged  to  be  operating  at  the  next  lower  level. 

•  Proceed  incrementally  until  the  team  reaches  a  point  at  which  all  of  the 
relevant  goals  are  rated  as  satisfied,  or  goal  ratings  lead  to  capability  level  0. 


Implementation  The  presence  of  advanced  practices  in  the  Engineering  PAs  creates  a  nuance 
Guidance  in  the  rating  process  that  can  be  complicated  for  some  appraisal  team 

members.  If  team  members  have  only  worked  with  the  staged  representation 
in  the  past,  it  is  important  that  the  appraisal  team  leader  covers  this  nuance 
during  team  training,  and  prevents  conftision  during  the  rating  process. 

Goal  satisfaction  is  a  judgment  based  on  the  implementation  of  practices  that 
map  to  the  goal.  In  rating  the  satisfaction  of  specific  goals  in  the  Engineering 
PAs,  the  set  of  specific  practices  that  relates  to  the  goals  differs  for  capability 
levels  0,  1,  2,  and  3  through  5.  That  is,  depending  on  the  capability  level  at 
which  the  rating  is  performed,  there  are  up  to  4  unique  sets  of  specific 
practices  associated  with  these  specific  goals  that  must  be  considered. 

The  appraisal  team  leader  is  responsible  for  selecting  one  of  the  two  optional 
rating  approaches  described  above,  and  should  facilitate  this  session  carefully 
to  prevent  confusion  among  team  members. 


♦ 
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2.4.2a  Determine  Process  Area  Capability  Level 
GSS  and  CPM  Implementation  Guidance 


No  additional  guidance. 
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2.4.2b  Determine  Satisfaction  of  Process  Areas 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Optional 

Practices 


Implementation 

Guidance 


When  using  the  staged  representation  of  a  CMMI  model,  the  team  may  derive 
the  satisfaction  of  PAs  from  the  set  of  goal  satisfaction  judgments.  Assigning 
PA  satisfaction  ratings  is  an  optional  activity,  selected  at  the  discretion  of  the 
appraisal  sponsor  and  documented  in  the  appraisal  input. 


Rate  the  satisfaction  of  each  PA  in  the  scope  of  the  appraisal  based  on  the 
ratings  of  the  goals  within  each  PA,  if  this  rating  option  was  selected  during 
planning. 


PAs  may  be  assigned  rating  values  of  Satisfied,  Unsatisfied,  Not  Applicable, 
or  Not  Rated. 

A  PA  is  rated  Satisfied  if  and  only  if  all  of  its  specific  goals  and  generic  goals 
are  rated  Satisfied. 

If  even  one  of  the  goals  in  a  PA  is  rated  Unsatisfied,  then  the  PA  is  rated 
Unsatisfied. 

When  a  PA  is  determined  to  be  outside  of  the  organizational  unit’s  scope  of 
work,  the  PA  is  designated  as  Not  Applicable  and  is  not  rated.  The 
identification  of  a  PA  as  Not  Applicable  must  occur  during  the  planning  of  the 
appraisal. 

When  a  PA  is  outside  of  the  appraisal  scope,  or  if  the  associated  set  of 
objective  evidence  does  not  meet  the  defined  criteria  for  sufficient  data 
coverage,  the  PA  is  designated  as  Not  Rated  and  is  not  rated.  The  criteria  for 
sufficient  data  coverage  are  described  in  activity  2.2.1,  Verify  Objective 
Evidence. 


A  profile  to  summarize  the  satisfaction  of  goals  may  be  created  to  provide 
further  insight  about  the  rating  outcomes.  Where  a  PA  is  rated  as  Unsatisfied, 
this  more  detailed  view  of  the  rating  outcomes  may  provide  focus  and 
visibility  at  a  lower  level  of  detail. 


PA  satisfaction  is  a  direct  function  of  goal  satisfaction.  A  PA  is  rated  as 
Satisfied  if  every  goal  contained  in  the  PA  is  rated  as  Satisfied.  A  PA  is  rated 
as  Unsatisfied  if  any  goal  is  rated  as  Unsatisfied.  This  ensures  that  one  or 
more  weaknesses  exist  that  serve  to  explain  why  the  goal  and  therefore  the  PA 
are  not  satisfied. 

PA  ratings  need  not  be  reported  to  appraisal  participants,  if  the  sponsor  does 
not  wish  to  disclose  these  results.  However,  a  documented  output  from  this 
rating  activity,  if  it  is  performed,  is  a  required  component  in  the  Appraisal 
Record. 


♦ 
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2.4.2b  Determine  Satisfaction  of  Process  Areas 


GSS  and  CPM  implementation  Guidance 

No  additional  guidance. 
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2.4.3a  Determine  Capability  Profile 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Optional 

Practices 


Implementation 

Guidance 


When  using  the  continuous  representation  of  a  CMMI  model,  the  team  may 
determine  a  Capability  Profile  that  graphically  depicts  the  capability  level 
ratings  assigned  to  each  PA  within  the  scope  of  the  appraisal.  The  generation 
of  a  Capability  Profile  is  an  optional  activity,  selected  at  the  discretion  of  the 
appraisal  sponsor  and  documented  in  the  appraisal  input. 


Generate  a  Capability  Profile  depicting  the  capability  level  attained  for  each 
PA  within  the  scope  of  the  appraisal,  if  this  rating  option  was  selected  during 
planning. 


A  simple  bar  chart  is  used  for  this  display.  Each  PA  is  represented  in  a  single 
bar  along  the  horizontal  axis,  and  the  vertical  axis  represents  the  capability 
level  dimension.  The  height  of  each  bar  communicates  the  capability  level  of 
the  PA  represented. 

Capability  levels  take  only  the  values  0,  1,  2,  3,  4,  or  5.  Intermediate  values 
(e.g.,  2.7)  are  not  defined  for  this  appraisal  outcome,  and  any  embellishment 
of  the  Capability  Profile  with  such  values  is  outside  the  boundaries  of 
SCAMPI. 


A  profile  to  summarize  the  satisfaction  of  goals  may  be  created  to  provide 
further  insight  about  the  rating  outcomes.  In  situations  where  a  PA  capability 
level  rating  does  not  reflect  the  desired  outcome,  this  more  detailed  view  may 
provide  focus  and  visibility  at  a  lower  level  of  detail. 

CMMI  provides  for  equivalent  staging,  whereby  a  Capability  Profile  can  be 
used  to  derive  an  equivalent  maturity  level  rating  (see  activity  2.4.3b, 
Determine  Maturity  Level). 


A  presentation  template  referred  to  as  a  Capability  Profile  is  typically  used  to 
communicate  the  aggregate  level  rating  results  to  the  sponsor  and  others 
designated  by  the  sponsor. 

Comparing  different  PAs  with  respect  to  their  relative  capability  level  ratings 
may  be  informative  in  discussing  trends  or  patterns  in  the  organization. 

This  activity  may  be  omitted  entirely,  as  it  is  a  tailoring  option.  If  a  Capability 
Profile  is  to  be  derived,  the  ratings  reflected  in  the  profile  are  derived  as 
described  in  activity  2.4.2b,  Determine  Process  Area  Capability  Level. 


♦ 
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2.4.3a  Determine  Capability  Profile 
GSS  and  CPM  Implementation  Guidance 


No  additional  guidance. 
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2.4.3b  Determine  Maturity  Level 


Activity 

Description 


Required 

Practices 

Parameters 
and  Limits 


Optional 

Practices 

Implementation 

Guidance 


Historically,  one  of  the  most  visible  outcomes  of  an  appraisal  has  been  the 
maturity  level  rating  assigned.  The  determination  of  a  maturity  level  rating  is 
straightforward,  and  is  derived  mechanically  from  the  ratings  assigned  at  the 
lower  levels  of  detail.  Assigning  a  maturity  level  rating  is  an  optional  activity, 
selected  at  the  discretion  of  the  appraisal  sponsor  and  documented  in  the 
appraisal  input. 


Rate  the  maturity  level  based  on  the  ratings  assigned  to  PAs,  if  this  rating 
option  was  selected  during  planning. 


The  maturity  level  determined  is  the  highest  level  at  which  all  PAs  contained 
within  the  maturity  level,  and  within  all  lower  maturity  levels,  are  rated  as 
Satisfied  or  Not  Applicable.  The  single  exception  to  this  rule  is  that  the 
maturity  level  3  goal  for  each  applicable  maturity  level  2  PA  must  also  be 
rated  Satisfied  for  a  maturity  level  rating  of  3  or  higher  to  be  determined. 

When  using  continuous  representations,  CMMI  provides  for  equivalent 
staging,  whereby  a  Capability  Profile  can  be  used  to  derive  an  equivalent 
maturity  level  rating.  A  maturity  level  for  a  continuous  representation  is 
achieved  if  the  Capability  Profile  is  at  or  above  the  target  profile  for  all  PAs 
for  that  maturity  level  and  all  lower  maturity  levels  in  the  equivalent  staging, 
excepting  those  PAs  that  are  designated  as  Not  Applicable.  The  equivalence 
of  particular  Capability  Profiles  and  particular  maturity  levels  is  addressed  in 
an  appendix  to  the  CMMI  model. 

To  determine  a  maturity  level  as  an  output  of  the  appraisal,  the  model  scope  of 
the  appraisal  must  include  the  minimum  set  of  PAs  required  by  the  CMMI 
model.  Please  refer  to  the  tailoring  section  of  the  CMMI  model  for  guidelines 
on  what  the  minimally  acceptable  scope  of  the  model  is  for  each  maturity 
level. 


-None. 


This  activity  may  be  omitted  entirely,  as  it  is  a  tailoring  option.  If  a  maturity 
level  is  to  be  reported,  the  PA  ratings  that  form  the  basis  for  the  maturity  level 
rating  are  derived  as  described  in  activity  2.4.2b,  Determine  Satisfaction  of 
Process  Areas. 


♦ 
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2.4.3b  Determine  Maturity  Level 
GSS  and  CPM  Implementation  Guidance 


No  additional  guidance. 
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2.4.4  Document  Appraisal  Results 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Optional 

Practices 

Implementation 

Guidance 


The  results  of  the  appraisal  conduct  must  be  documented  for  reporting.  Verbal 
reports  of  the  rating  outcomes  or  face-to-face  explanations  of  implementation 
gaps  discovered  by  the  team  are  not  sufficient  to  communicate  appraisal 
results. 


•  Document  the  final  findings. 

•  Document  the  rating  outcome(s). 

•  Document  the  Appraisal  Disclosure  Statement  (ADS). 


The  ADS  and  the  set  of  appraisal  outputs  agreed  upon  with  the  appraisal 
sponsor  must  be  documented.  These  appraisal  outputs  may  exclude  all  ratings, 
and  the  sponsor  is  free  to  select  and  disclose  a  variety  of  appraisal  outcomes, 
as  specified  in  the  activities  of  this  process. 

Regardless  of  the  needs  of  the  sponsor,  the  ADS,  the  goal  ratings,  and  the 
associated  findings  must  be  documented  as  a  part  of  the  appraisal  information 
returned  to  the  CMMI  Steward. 


Any  optional  outputs  requested  by  the  appraisal  sponsor  are  also  created 
during  this  activity. 


This  activity  is  focused  on  collecting  and  documenting  the  results  of  prior 
activities  related  to  the  generation  of  findings  and  ratings.  Depending  on  the 
planned  recipients  of  the  results,  multiple  forms  of  the  results  may  be  needed. 
Certain  data  may  not  be  appropriate  for  all  audiences,  or  the  style  or  language 
of  the  results  may  need  to  be  adjusted  to  best  fit  the  needs  of  the  recipients. 

The  documented  appraisal  results  are  typically  provided  in  a  final  findings 
presentation,  described  in  activity  3.1.1,  Present  Final  Findings. 


♦ 
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2.4.4  Document  Appraisal  Results 


GSS  and  CPM  Implementation  Guidance 

No  additional  guidance. 
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3.1  Deliver  Appraisal  Results 


Purpose 


Entry  Criteria 


Inputs 


Activities 


Outputs 


Outcome 


Exit  Criteria 


Provide  credible  appraisal  results  that  can  be  used  to  guide  actions.  Represent 
the  strengths  and  weaknesses  of  the  processes  in  use  at  the  time.  Provide 
ratings  (if  planned  for)  that  accurately  reflect  the  capability  level  or  maturity 
level  of  the  processes  in  use. 


•  Objective  evidence  has  been  validated  (through  the  team  process). 

•  Preliminary  findings  have  been  validated. 

•  Ratings  have  been  determined  (for  model  components  selected  for  rating). 

•  Final  findings  have  been  created  and  reviewed  by  the  team. 


•  Appraisal  data 

final  findings 
ratings 

•  Appraisal  artifacts 

appraisal  input 
appraisal  plan 


3.1.1  Present  Final  Findings 

3.1.2  Conduct  Executive  Session(s) 

3.1.3  Plan  for  Next  Steps 


•  Documented  final  findings 

•  Final  report  (if  requested) 

•  Recommendations  report  (if  requested) 


•  The  sponsor  and  the  appraised  organizational  unit  are  provided  with  the 
results  of  the  appraisal. 

•  A  valid  and  reliable  characterization  of  the  current  state  of  the  processes 
in  use  across  the  organizational  unit  is  documented. 


•  Appraisal  results  are  delivered  to  the  appraisal  sponsor  and  organizational 
unit. 

•  An  executive  session  is  conducted,  if  appropriate. 


Continued  on  next  page 
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3.1  Deliver  Appraisal  Results  (continued) 


Key  Points 


Tools  and 
Techniques 


Metrics 


Verification  and 
Validation 


Records 


Tailoring 


The  appraisal  results  are  intended  to  support  decision  making,  and  need  to  be 
delivered  in  a  way  that  promotes  appropriate  actions.  Whether  the  appraisal 
was  conducted  for  internal  process  improvement,  supplier  selection,  or 
process  monitoring  purposes,  the  delivery  of  results  should  facilitate  the 
actions  that  will  be  driven  by  the  information. 


Templates  containing  standard  information  for  use  in  a  final  findings  briefing 
are  provided  to  all  SCAMPI  Lead  Appraisers.  Experienced  appraisers 
frequently  use  electronic  (database)  tools  that  support  the  transformation  of 
raw  appraisal  data  into  appraisal  results.  These  tools  may  be  useful  in  real 
time  as  appraisal  results  are  presented.  Strategies  for  presenting  and  packaging 
the  results  should  leverage  presentation  and  documentation  techniques  that 
best  suit  the  audience. 


It  is  highly  recommended  that  the  attendance  at  the  final  briefing  (if  one  is 
held)  be  recorded.  Significant  absenteeism  of  key  stakeholders  is  likely  to  be 
an  indication  of  risk  for  future  success  in  addressing  the  appraisal  findings. 


The  required  elements  of  appraisal  results  are  specified  in  the  activity 
description  found  here,  and  a  checklist  can  support  verification  that  these 
elements  are  present.  Validation  of  this  activity  can  only  occur  after  the 
appraisal  is  complete. 


•  Final  findings 

•  Final  report  (if  requested) 

•  Recommendations  report  (if  requested) 


If  the  method  is  being  used  as  part  of  a  supplier  selection  process,  there  may 
be  acquisition  regulations  or  limitations  that  constrain  the  mechanisms  used  to 
deliver  appraisal  results  to  the  appraised  organization. 

In  some  internal  process  improvement  usage  of  the  method,  the  executive 
session  may  be  tailored  out.  The  appraisal  sponsor  should  make  this  decision, 
with  the  full  involvement  of  the  appraisal  team  leader. 


Continued  on  next  page 
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3.1  Deliver  Appraisal  Results  (continued) 


Interfaces 
with  Other 
Processes 


Summary  of 
Activities 


Upon  completion  of  the  Generate  Appraisal  Results  process,  the  ratings  and 
findings  generated  are  used  to  prepare  and  deliver  the  final  appraisal  results  to 
the  appraisal  sponsor  and  organizational  unit.  The  appraisal  results  become 
part  of  the  Appraisal  Record,  which  is  discussed  in  process  3.2,  Package  and 
Archive  Appraisal  Assets. 


The  final  findings  contain  the  validated  strengths,  weaknesses,  and  ratings  (as 
defined  by  the  appraisal  plan),  reflecting  the  organizational  process  capability 
and/or  maturity  level  for  PAs  within  the  appraisal  scope.  Other  appraisal 
outputs,  as  requested  by  the  appraisal  sponsor  and  documented  in  the  appraisal 
plan,  are  generated  and  provided.  Optionally,  a  separate  executive  session 
may  also  be  held  to  clarify  and  discuss  the  appraisal  results  from  a  senior 
management  perspective  that  facilitates  decision  making.  Plans  are  established 
for  acting  upon  the  appraisal  results. 


♦ 
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3.1  Deliver  Appraisal  Results 


GSS  and  CPM  Implementation  Guidance 

When  using  the  MDD  for  Government  source  selection  and  contract  process  monitoring,  the 
following  should  be  considered: 

Guidance  for  Findings  should  always  be  delivered  at  the  earliest  possible  time  within  GSS 
GSS  and  CPM  and  CPM  constraints. 

In  a  source  selection  or  contract  process  monitoring  SCAMPI,  the  results  are 
not  “confidential,”  in  that  the  sponsor  is  an  outside  organization  from  the 
appraised  organization.  However,  the  results  are  only  known  to  the  sponsor 
and  the  appraised  organization.  Competing  organizations  do  not  see  the 
results. 

Guidance  for  Although  delivery  of  the  final  findings  to  the  development  organization  is 
GSS  required  by  the  SCAMPI  method,  contractual  and  legal  constraints  may 

preclude  full  execution  of  this  activity  until  after  contract  award.  Instead,  the 
final  meeting  at  the  conclusion  of  the  site  visit  may  be  a  “thank  you”  exit 
briefing.  At  a  minimum,  the  SCAMPI  team  members  should  thank  their  hosts 
and  provide  some  indication  of  when  the  findings  results  and  information 
about  the  individual  development  organization’s  performance  would  be 
available,  who  to  contact,  and  how  to  proceed. 

Appraisal  results  are  incorporated  into  the  evaluation  context  to  be  presented 
to  the  Source  Selection  Advisory  Council  (SSAC)  and  Source  Selection 
Authority  (SSA).  The  SSEB  compares  data  collected  against  evaluation 
standards  and  assigns  technical  ratings  and  risk  identifications.  The  SSAC 
reviews  the  SSEB  evaluation  results  and  presents  the  results  to  the  SSA.  The 
SSA  makes  the  award  decision.  The  SCAMPI  team  may  or  may  not  be 
formally  part  of  the  SSEB.  If  they  are  not,  the  team  provides  SCAMPI 
findings  and  outcomes  to  the  SSEB.  The  team  consults  with  the  SSEB  if 
requested.  The  team  may  act  as  an  advisor  to  the  SSAC  and  SSA. 

Guidance  for 
CPM 


♦ 


In  a  CPM  appraisal,  the  final  findings  can  usually  be  delivered  at  the 
conclusion  of  the  site  visit.  However,  competitive  incentives  among  multiple 
suppliers  might  delay  delivery  of  final  findings.  Providing  the  “sanitized” 
(personnel  references  removed)  PH  sheets  or  database  to  the  appraised 
organization  may  be  useful  to  initiate  or  support  process  improvement, 
particularly  in  a  collaborative  acquirer/supplier  application. 
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3.1.1  Present  Final  Findings 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


The  final  findings  contain  a  summary  of  the  strengths  and  weaknesses  for 
each  PA  within  the  appraisal  scope,  as  well  as  additional  information  that 
provides  context  for  the  findings.  The  generation  of  the  findings  is  addressed 
in  activity  2.4.1,  Derive  Findings  and  Rate  Goals;  this  activity  relates  to  the 
presentation  of  these  findings  to  the  appraisal  sponsor  and  appraised 
organization.  The  presentation  may  be  in  a  summarized  form,  with  the 
detailed  findings  provided  as  backup  information,  and  is  often  presented  using 
view  graphs  in  a  meeting  room  or  auditorium. 

In  addition  to  the  final  findings,  a  draft  ADS  summarizing  the  results  of  the 
appraisal  is  provided  to  the  appraisal  sponsor. 


•  Provide  appraisal  final  findings  to  the  appraisal  sponsor  and  the 
organizational  unit. 

•  Provide  an  ADS  to  the  appraisal  sponsor  summarizing  the  appraisal 
results  and  conditions  under  which  the  appraisal  was  performed. 


Required  elements  of  the  final  findings  include 

•  summary  of  the  appraisal  process 

•  findings  (summary  of  strengths  and  weaknesses) 

Appraisal  team  consensus  must  be  obtained  on  the  wording  of  the  final 
findings,  to  ensure  that  the  whole  team  supports  the  accuracy  of  the  described 
appraisal  results. 

The  team,  when  delivering  the  final  findings,  must  adhere  to  some  important 
principles: 

•  If  a  model  component  is  reported  as  Unsatisfied,  the  corresponding 
findings  of  weaknesses  that  caused  the  team  to  make  that  judgment  must 
also  be  reported. 

•  Confidentiality  and  non-attribution  principles  apply  to  statements  made  in 
the  presentation  of  final  findings. 

The  ADS  is  a  summary  statement  describing  the  appraisal  results  that  includes 
the  conditions  and  constraints  under  which  the  appraisal  was  performed.  It 
contains  information  considered  essential  to  adequately  interpret  the  meaning 
of  assigned  maturity  level  or  capability  level  ratings.  The  ADS  is  prepared  by 
the  appraisal  team  leader  and  provided  to  the  appraisal  sponsor.  Otherwise  the 
appraisal  team  leader  delivers  the  ADS  to  the  sponsor  as  a  separate  document. 

A  detailed  description  of  the  ADS  contents  is  provided  in  Appendix  A.  The 
ADS  is  considered  a  draft  at  this  stage  of  the  appraisal  process,  in  that  the 
ADS  must  also  contain  an  affirmation  that  all  appraisal  requirements  have 
been  satisfied,  which  cannot  be  claimed  until  the  completion  of  all  appraisal 
activities. 


Continued  on  next  page 
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3.1.1  Present  Final  Findings  (continued) 


Optional 

Practices 


Optional  elements  of  the  final  findings  include 

•  ratings 

•  improvement  activities 

•  recommended  actions 

•  schedule  of  major  upcoming  events  (e.g.,  appraisal  report, 
recommendations,  action  plan,  reappraisal) 

Note  that  the  generation  of  goal  ratings  by  the  appraisal  team  is  required  (as 
described  in  process  2.4,  Generate  Appraisal  Results).  However,  these  ratings 
may  be  excluded  from  the  final  findings  at  the  discretion  of  the  appraisal 
sponsor. 

A  formal  presentation  of  appraisal  results,  delivered  by  the  appraisal  team,  is 
frequently  the  final  visible  activity  for  appraisals  conducted  for  internal 
process  improvement.  The  final  findings  presentation  typically  is  delivered  in 
the  form  of  a  face-to-face  briefing  at  the  end  of  the  appraisal  on-site  period. 
Other  mechanisms  for  providing  the  appraisal  results  to  the  organizational 
unit,  such  as  written  reports,  may  be  more  practical  in  supplier  selection  or 
process  monitoring  usage  of  the  method.  The  timeframe  in  which  the  appraisal 
results  are  provided  may  also  vary,  but  the  appraisal  cannot  be  considered 
complete  until  the  final  findings  are  provided. 

The  draft  ADS  may  optionally  be  provided  during  the  executive  session(s),  if 
performed,  instead  of  at  the  conclusion  of  the  final  findings  briefing. 


Continued  on  next  page 
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3.1 .1  Present  Final  Findings  (continued) 


Implementation 

Guidance 


A  template  for  a  final  findings  briefing,  describing  its  typical  contents  and 
format,  is  provided  to  Lead  Appraisers  as  a  work  aid  by  the  CMMI  Steward. 

Findings  include  a  summary  of  strengths  and  weaknesses  determined  for  each 
PA  within  the  appraisal  reference  model  scope.  This  may  also  include  global 
findings  that  apply  across  multiple  PAs,  and  non-reference  model  findings 
that  affect  the  implementation  (positively  or  negatively)  of  associated 
processes  within  the  organizational  unit. 

Normally,  the  appraisal  team  leader  presents  the  final  findings.  In  some 
applications  of  the  method  for  internal  process  improvement,  the  team  may 
elect  to  have  an  appraisal  team  member  from  the  organizational  unit  provide 
the  briefing  to  encourage  the  acceptance  of  the  final  findings  and  ownership  of 
the  appraisal  results  for  follow-on  action. 

As  a  courtesy,  the  appraisal  team  can  consider  informing  the  appraisal  sponsor 
and/or  the  senior  site  manager  of  the  appraisal  results  prior  to  presenting  them 
publicly  in  the  final  findings  briefing.  This  may  help  them  to  avoid  surprises 
and  obtain  feedback  on  ways  to  present  the  findings  that  best  meet  the  needs 
of  the  sponsor,  appraisal  participants,  and  the  organizational  unit.  See  activity 
3.1.2,  Conduct  Executive  Session(s)  for  a  description  of  topics  for  discussion. 

The  number  and  scope  of  findings  reported  will  affect  the  impact  of  appraisal 
results,  whether  or  not  the  team  intends  for  this  to  happen.  There  are  times 
when  providing  a  long  list  of  details  is  beneficial.  Other  times,  high-level 
summaries  are  more  appropriate. 


♦ 
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3.1.1  Present  Final  Findings 


GSS  and  CPM  Implementation  Guidance 

When  using  the  MDD  for  Government  source  selection  and  contract  process  monitoring,  the 
following  should  be  considered: 

Guidance  for  For  determination  of  the  reporting  of  appraisal  results,  see  activity  1.1.2, 
GSS  and  CPM  Determine  Appraisal  Constraints,  and  activity  1.1.4,  Determine  Outputs. 

Guidance  for  There  may  be  source  selection  constraints  (e.g.,  whether  discussions  have 

GSS  been  opened)  bearing  on  the  ability  to  present  final  findings  until  after 

contract  award.  In  a  source  selection  environment,  a  written  report  may  be 
used  in  lieu  of  a  face-to-face  briefing  when  final  findings  are  not  provided 
during  the  on  site.  Final  findings  should  be  delivered  as  soon  as  possible 
within  the  source  selection  schedule.  The  appraisal  is  not  complete  until  the 
final  findings  are  provided.  The  final  findings  report  should  provide  details  on 
appraisal  findings.  It  forms  the  basis  of  follow-on  actions  and  may  be  used  for 
input  in  subsequent  appraisals  to  determine  model  scope.  Face-to-face 
briefings  could  be  included  in  the  source  selection  out-briefs  to  the  successful 
and  unsuccessful  offerors. 

In  some  source  selections,  technical  discussions  are  defined  as  the  period  after 
final  submission  of  the  cost  proposal.  An  alternative  approach  is  to  conduct  a 
“split  proposal,”  in  which  the  offeror  is  asked  to  provide  the  technical 
proposal  separate  from  cost.  The  SCAMPI  appraisal  is  performed  prior  to 
receipt  of  the  cost  proposal,  during  the  discovery  part  of  the  source  selection. 
This  allows  time  for  technical  dialog  regarding  SCAMPI  findings  prior  to 
formal  discussions.  Two  out-briefs  are  conducted.  In  the  first,  held  prior  to  the 
cost  proposal  review,  the  offeror  is  given  a  chance  to  respond  to  weaknesses 
identified  as  part  of  the  appraisal.  This  first  out-brief  serves  to  validate 
preliminary  findings.  The  second  out-brief,  the  final  findings,  is  provided  at 
contract  award. 

Acquisition/procurement  restrictions  may  shift  the  time  frame  for  reporting 
results  to  the  CMMI  Steward.  The  appraisal  ADS  may  not  be  forwarded  in  the 
nominal  30-60  days  following  the  on  site(s)  of  offeror(s).  Consequently,  the 
reporting  of  the  final  findings  to  the  appraised  entity  may  be  delayed  until 
after  contract  award. 

Note:  This  is  another  example  of  an  approach  to  satisfy  the  SCAMPI 
requirement  discussed  in  activity  2.2.3,  Validate  Practice  Implementation 
Gaps. 

Guidance  for  The  time  frame  for  reporting  results  to  the  sponsor,  the  appraised 

CPM  organization,  and  the  CMMI  Steward  may  be  shifted  due  to  acquisition/ 

procurement  restrictions,  although  this  is  usually  not  an  issue  in  CPM. 
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3.1 .2  Conduct  Executive  Session(s) 


Activity 

Description 


Required 

Practices 

Parameters 
and  Limits 


The  executive  session  is  an  optional  activity  that  may  be  performed  at  the 
discretion  of  the  appraisal  sponsor  or  senior  site  manager.  The  executive 
session  provides  the  appraisal  sponsor,  senior  site  manager,  and  invited  staff  a 
private  opportunity  to  (a)  discuss  with  the  appraisal  team  leader  any  issues 
with  the  appraisal,  (b)  obtain  clarification  of  the  appraisal  results,  (c)  confirm 
understanding  of  the  process  issues,  and  (d)  provide  guidance  regarding  focus, 
timing,  and  priorities  of  the  recommendations  report  and  follow-on  activities. 


None.  If  the  option  is  selected,  hold  a  private  meeting  between  the  appraisal 
team  leader  and  the  sponsor,  including  any  participants  invited  by  the  sponsor. 


If  an  executive  session  is  conducted,  the  confidentiality  and  non-attribution  of 
data  sources  must  be  maintained. 

Multiple  sessions  may  be  held  if  necessary,  targeted  at  the  information  needs 
of  the  executive  audience. 


Continued  on  next  page 
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3.1.2  Conduct  Executive  Session(s)  (continued) 


Optional 

Practices 


Implementation 

Guidance 


Page  11-250 


Attendance  by  the  entire  appraisal  team  at  the  executive  sessions  is  a  tailoring 
option. 

The  executive  session  is  also  an  appropriate  opportunity  to  review  appraisal 
performance  with  the  appraisal  sponsor  and/or  senior  site  manager,  and 
planned  versus  actual  execution  of  the  appraisal  plan,  including  method 
tailoring.  This  provides  additional  input  on  the  appropriate  expectations  for 
interpreting  and  handling  the  appraisal  results. 

The  draft  ADS  may  optionally  be  provided  during  the  executive  session 
instead  of  at  the  conclusion  of  the  final  findings  briefing  as  discussed  in 
activity  3.1.1,  Present  Final  Findings. 


The  intent  of  the  executive  sessions  is  to  ensure  that  the  appraisal  sponsor 
and/or  the  senior  site  manager  have  a  sound  understanding  of  the  appraisal 
results.  Any  feedback  obtained  from  these  executive  sessions  should  be 
recorded.  All  rules  for  confidentiality  and  non-attribution  are  still  in  effect. 


♦ 
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3.1 .2  Conduct  Executive  Session(s) 

GSS  and  CPM  Implementation  Guidance 

When  using  the  MDD  for  Government  source  selection  and  contract  process  monitoring,  the 
following  should  be  considered: 

Guidance  for  In  a  GSS  environment  and/or  when  briefing  of  final  findings  takes  place  after 
GSS  the  on-site  period,  multiple  executive  sessions  may  be  required.  The  initial 

session  serves  to  brief  the  sponsor  (SSEB)  prior  to  contract  award.  Multiple 
executive  sessions  may  be  held  after  contract  award  to  brief  the  on-site 
managers  of  the  successful  and  unsuccessful  offerors  on  the  results  of  the 
appraisal.  A  more  in-depth  session  would  be  held  with  the  successful  offerors 
to  ensure  understanding  of  weaknesses  and  the  priority  of  risk-mitigation 
activities. 

Guidance  for  No  additional  guidance. 

CPM 

♦ 
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3.1 .3  Plan  for  Next  Steps 


Activity 

Description 


Required 

Practices 

Parameters 
and  Limits 

Optional 

Practices 

Implementation 

Guidance 


implementation 

Guidance 

Process 
Improvement 
Action  Planning 


Following  the  delivery  of  the  appraisal  results,  a  plan  for  follow-on  activities 
is  determined.  The  planned  follow-on  activities  are  typically  defined  in  the 
appraisal  plan,  reflecting  sponsor  requests  for  additional  appraisal  tasks  and 
products  necessary  to  meet  appraisal  objectives,  or  for  a  commitment  to  take 
action  upon  the  appraisal  results.  Follow-on  activities  may  include 

•  development  of  a  final  report 

•  development  of  a  recommendations  report  or  briefing 

•  generation  or  update  of  a  process  improvement  plan 


None. 


None. 


Planning  for  next  steps  is  an  optional,  but  recommended,  appraisal  activity. 


Planning  for  next  steps  includes  optional  activities  such  as 

•  development  of  a  final  report  by  the  appraisal  team,  summarizing  the 
appraisal  results  for  delivery  to  the  appraisal  sponsor 

•  submission  of  appraisal  team  recommendations  for  action  upon  the 
appraisal  findings 

•  generation  of  a  process  improvement  action  plan  for  the  organizational 
unit  to  act  upon  the  appraisal  findings 

In  addition  to  specifying  the  activities  to  be  performed,  these  plans  usually 
include  the  assignment  of  responsibility,  schedule,  and  estimated  resources  for 
the  implementation  of  the  follow-on  actions.  The  plans  established  can  be 
used  to  track  the  progress  of  the  follow-on  activities  over  time. 


Findings  and  recommendations  from  the  appraisal  team  can  be  used  by  the 
Organizational  unit  to  establish  action  plans  for  process  improvement.  This  is 
an  optional  output  most  often  used  in  internal  process  improvement  or 
process-monitoring  applications  of  the  appraisal  method. 

Recommendations  often  include  a  prioritized  list  of  improvement  activities, 
including  the  development  of  an  improvement  plan  that  defines  the  tasks, 
schedules,  and  resources  necessary  for  implementation. 

Follow-on  appraisals  are  usually  performed  to  verify  improvement  progress. 
This  might  include  a  combination  of  Class  A,  Class  B,  and  Class  C  appraisals 
(refer  to  the  ARC  for  additional  details). 


Continued  on  next  page 
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3.1 .3  Plan  for  Next  Steps  (continued) 


Implementation 

Guidance 

Final  Report 


Implementation 

Guidance 

Recommend¬ 
ations  Report 


The  purpose  of  the  final  report  is  to  provide  details  or  explanations  beyond 
what  was  contained  in  the  final  findings.  The  generation  of  an  appraisal  final 
report  is  an  optional  activity  that,  if  requested  by  the  appraisal  sponsor, 
documents  the  execution  of  the  appraisal,  contains  detailed  appraisal  findings, 
and  forms  a  basis  for  action  planning.  This  baseline  is  used  for  subsequent 
reports  and  follow-on  actions,  and  also  may  be  an  input  for  use  in  subsequent 
appraisals. 

Items  contained  or  referenced  in  the  final  report,  either  in  their  entirety  or  as  a 
subset,  might  include 

•  executive  summary  of  the  appraisal  process  and  results 

•  appraisal  input  (see  process  1.1) 

•  appraisal  plan  (see  process  1.2) 

•  appraisal  record  (see  process  3.2) 

The  final  report  should  be  completed  as  soon  after  the  appraisal  as  possible, 
preferably  within  four  weeks.  The  appraisal  team  leader  usually  generates  the 
final  report;  other  team  members  may  also  contribute. 

The  format  and  content  of  the  final  report  may  vary  according  to  its  intended 
use  by  the  appraisal  sponsor.  In  its  simplest  form,  this  could  be  a  set  of  notes 
annotated  to  the  final  findings,  elaborating  on  some  aspect  of  the  findings  or 
capturing  essential  comments  or  recommendations  from  the  appraisal  team. 


If  requested  by  the  appraisal  sponsor,  appraisal  team  recommendations  for 
taking  action  on  the  appraisal  results  can  be  provided.  These 
recommendations  can  provide  momentum  to  the  appraisal  follow-up  by 
serving  as  a  link  between  the  appraisal  findings  and  subsequent  decision 
making  or  action  plans.  The  emphasis  of  these  recommendations  depends  on 
the  appraisal  sponsor’s  objectives  and  planned  use  of  the  appraisal  results,  as 
defined  in  the  appraisal  input.  This  can  vary  widely  based  on  the  context  in 
which  the  appraisal  method  is  applied  (e.g.,  internal  process  improvement, 
supplier  .selection,  process  monitoring). 

The  recommendations  report  should  be  completed  as  soon  after  the  appraisal 
on-site  period  as  possible.  Depending  on  the  nature,  complexity,  and  use  of 
the  recommendations,  this  may  take  as  long  as  two  months  to  produce. 

Rather  than  generate  a  separate  recommendations  report,  a  common 
alternative  is  to  include  these  recommendations  in  the  final  report. 

It  is  important  to  consider  the  possibility  that  the  expertise  needed  for  making 
the  appropriate  recommendations  may  be  beyond  the  level  of  expertise 
reflected  on  the  team. 
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3.1 .3  Plan  for  Next  Steps 


GSS  and  CPM  Implementation  Guidance 

When  using  the  MDD  for  Government  source  selection  and  contract  process  monitoring,  the 
following  should  be  considered: 

Guidance  for  Next  steps  may  include  planning  a  date  for  delivery  of  final  findings  either  by 

GSS  person  or  written  report.  Next  steps  for  the  successful  offerors  should  include 

a  requirement  to  develop  a  risk  management  plan  to  address  weaknesses,  with 
milestone  dates  for  completion  of  process  improvement  activities.  Milestone 
dates  could  be  tied  to  an  incentive  fee.  The  GSS  SCAMPI  appraisal 
establishes  a  “process  baseline,”  lending  further  utility  to  the  SCAMPI  in 
process  monitoring  for  considering  award  fees  or  value  engineering  incentives 
for  systems/software  process  improvement.  The  GSS  SCAMPI  findings  are 
used  to  frame  the  award  fee  plan.  Note,  however,  that  award  fee  applications 
(i.e.,  an  award  for  meeting  specified  measures  of  performance)  are  not 
appropriate  in  all  instances. 

Guidance  for  Next  steps  could  include  plans  for  subsequent  appraisals  to  monitor  progress 

CPM  against  a  risk  management  plan  that  addresses  weaknesses  identified  as  part  of 

the  baseline  appraisal.  GSS  final  findings  may  be  used  for  input  in  subsequent 
appraisals  to  determine  model  scope.  For  existing  contracts,  SCAMPI,  as  a 
contract  process  monitoring  tool,  can  be  used  as  a  negotiated  contractual 
action.  The  plan  should  address  the  type  of  subsequent  appraisal  planned 
(ARC  Class  A,  B,  or  C),  timing  of  appraisals  (periodic  or  milestone  driven), 
and  trading  off  the  surveillance  of  strong  areas  for  weak  ones.  ARC  Class  B  or 
C  appraisals  focusing  on  the  PAs  where  process  weaknesses  present  the  most 
risk  may  be  more  appropriate  for  monitoring  progress  towards  a  risk 
mitigation  plan.  Alternatives  for  subsequent  appraisal  teams  include  Program 
Management  Office-sponsored  teams,  in-plant  Government  representative 
(DCMA)  teams,  and  Government  participation  on  internal  process  appraisals. 
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3.2  Package  and  Archive  Appraisal  Assets 


Purpose 
Entry  Criteria 

Inputs 

Activities 

Outputs 

Outcome 

Exit  Criteria 


Preserve  important  data  and  records  from  the  appraisal,  and  dispose  of 
sensitive  materials  in  an  appropriate  manner. 


•  Appraisal  has  been  conducted. 

•  Results  have  been  delivered  to  the  sponsor. 

•  All  appropriate  data  have  been  collected  and  retained  during  the  appraisal. 


•  Appraisal  data 

appraisal  input 
appraisal  plan 
final  findings 
objective  evidence 

•  Appraisal  team  artifacts 

notes 

documented  practice  implementation  gaps 
preliminary  findings 
document  library 


3.2. 1  Collect  Lessons  Learned 

3.2.2  Generate  Appraisal  Record 

3.2.3  Provide  Appraisal  Feedback  to  CMMI  Steward 

3.2.4  Archive  and/or  Dispose  of  Key  Artifacts 


•  Appraisal  Record 

•  Completed  forms  and  checklists 

•  Sanitized  data  (as  appropriate  and  agreed  upon  during  planning) 

•  Lessons  learned  (appraisal  team,  organization) 


Data  and  artifacts  are  appropriately  archived  or  destroyed.  The  team  has 
captured  lessons  and  data  to  help  improve  the  appraisal  process.  Requirements 
for  providing  appraisal  artifacts  to  stakeholders  and  the  CMMI  Steward  are 
met. 


•  Appraisal  assets  are  baselined  and  archived. 

•  Required  reports  are  delivered  to  the  appropriate  stakeholders. 

•  Artifacts  containing  sensitive  information  are  disposed  of  in  an 
appropriate  manner. 


Continued  on  next  page 
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3.2  Package  and  Archive  Appraisal  Assets  (continued) 


Key  Points 

Tools  and 
Techniques 

Metrics 

Verification  and 
Validation 

Records 

Tailoring 


Protect  the  confidentiality  of  sensitive  data  while  distributing  and  archiving 
appraisal  assets.  Bundle  related  information  together  whenever  appropriate. 


The  use  of  electronic  (database)  tools  for  managing  appraisal  data  often 
provides  assistance  in  ensuring  the  integrity  of  baselines,  as  well  as 
repackaging  information  for  archival  purposes.  Electronic  tools  allow  the 
Lead  Appraiser  to  remove  traceability  information  so  that  data  can  be 
provided  to  the  appropriate  people  while  preserving  the  anonymity  of  the  data 
sources. 

Electronic  tools  also  support  the  submission  of  appraisal  data  to  the  CMMI 
Steward.  This  reduces  the  administrative  burden,  and  will  facilitate  the 
analysis  of  appraisal  method  performance  data.  These  tools  also  provide 
feedback  on  the  consolidated  analysis  results  to  the  appraisal  community. 


While  archiving  and  reporting  the  metrics  associated  with  the  conduct  of  the 
appraisal  is  an  important  element  of  this  activity,  the  metrics  associated  with 
the  conduct  of  this  activity  itself  are  limited.  The  effort  and  calendar  time 
consumed  are  collected  and  compared  to  the  plan.  Some  appraisal  team 
leaders  will  choose  to  maintain  personal  metrics  associated  with  the  artifacts 
described  in  this  activity. 


The  Lead  Appraiser  Requirements  Checklist  guides  the  verification  of  the  list 
of  artifacts  provided  to  the  CMMI  Steward.  Validation  is  provided  by  the 
CMMI  Steward  upon  receipt  of  the  appraisal  record. 


•  Appraisal  Record 

•  Lessons  Learned 


The  usage  mode  and  constraints  of  the  appraisal,  as  well  as  the  sensitivity  of 
the  data  and  planned  use  of  appraisal  results,  may  greatly  affect  the  degree  to 
which  appraisal  data  is  retained,  sanitized,  or  discarded. 


Continued  on  next  page 


Page  11-258 


CMU/SEI-2002-HB-002 


3.2  Package  and  Archive  Appraisal  Assets  (continued) 


Interfaces 
with  Other 
Processes 


Summary  of 
Activities 


As  the  final  process  in  the  appraisal,  this  process  is  about  collecting, 
packaging,  and  archiving  those  results  and  artifacts  produced  by  previous 
processes  that  must  become  part  of  the  appraisal  record.  Most  notably,  this 
includes  the  appraisal  input,  appraisal  plan,  and  appraisal  results. 
Additionally,  sensitive  or  proprietary  data  produced  by  other  appraisal 
processes  must  be  returned  to  the  organizational  unit  or  destroyed. 


This  process  performs  the  data  collection,  data  management,  and  reporting 
activities  necessary  to  close  out  the  appraisal.  Data  collected  throughout  the 
appraisal  is  consolidated  and  baselined,  becoming  a  permanent  part  of  the 
appraisal  record. 
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3.2  Package  and  Archive  Appraisal  Assets 
GSS  and  CPM  Implementation  Guidance 


No  additional  guidance. 
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3.2.1  Collect  Lessons  Learned 


Activity 

Description 


Required 

Practices 

Parameters 
and  Limits 


As  one  of  the  final  activities  in  wrapping  up  an  appraisal,  teams  typically 
record  lessons  learned  from  their  experience.  The  purpose  of  these  lessons 
learned  is  to  document  what  went  right,  what  went  wrong,  and  any 
suggestions  or  recommendations  for  improving  the  method  or  its  execution. 
The  collection  of  lessons  learned  is  a  recommended  activity  for  the 
improvement  of  future  appraisals,  but  is  not  a  method  requirement. 


None. 


Lessons  learned  must  adhere  to  the  same  principles  of  confidentiality  and  non¬ 
attribution  applicable  to  other  appraisal  results. 


Continued  on  next  page 
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3.2.1  Collect  Lessons  Learned  (continued) 


Optional 

Practices 


Implementation 

Guidance 


Page  11-264 


All  practices  related  to  the  collection  of  lessons  learned  are  optional,  but 
recommended.  If  the  team  has  identified  potential  improvements  to  elements 
of  the  CMMI  Product  Suite  (reference  model,  appraisal  method,  and  training 
materials),  these  can  be  submitted  as  change  requests  to  the  CMMI  Steward. 


Capturing  lessons  learned  is  often  done  as  a  group  at  the  end  of  the  appraisal, 
while  the  appraisal  activities  are  fresh  in  team  members’  minds.  This  can  be 
supplemented  with  additional  inputs  from  team  members  upon  further 
reflection,  if  necessary.  Appraisal  team  leaders  forward  these  aggregate 
lessons  learned,  as  appropriate,  to  various  stakeholders,  but  always  to  the 
other  team  members.  Team  leaders  and  members  often  maintain  summary 
lists  of  appraisal  best  practices  and  lessons  learned  as  a  mechanism  for 
continuous  learning  and  improvement,  and  these  are  used  as  a  resource  for 
planning  subsequent  appraisals. 


♦ 


CMU/SEI-2002-HB-002 


3.2.1  Collect  Lessons  Learned 


GSS  and  CPM  Implementation  Guidance 

When  using  the  MDD  for  Government  source  selection  and  contract  process  monitoring,  the 
following  should  be  considered: 

Guidance  for  Lessons  learned  should  be  captured  regarding  how  to  better  provide  findings 

GSS  (format  and  content)  that  are  meaningful  and  useful  to  a  source  selection, 

including  what  is  useful  and  not  useful  for  discriminating  among  the  offerors. 
Additional  lessons  learned  could  be  related  to  the  suggested  weighting  of 
findings/results. 

Guidance  for  Lessons  learned  should  be  captured  regarding  how  to  better  provide  findings 

CPM  (format  and  content)  that  are  meaningful  and  useful  to  contract  process 

monitoring,  including  what  is  useful  and  not  useful  for  determining  award  fees 
and  other  follow  on  actions.  Additional  lessons  learned  could  be  related  to  the 
suggested  weighting  of  findings/results  and  for  how  to  address  partial  or  delta 
appraisals  when  recommended  for  investigating  previously  determined 
weaknesses. 
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3.2.2  Generate  Appraisal  Record 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Appraisal  data  collected  throughout  the  appraisal  is  aggregated  and 
summarized  into  a  permanent  record  documenting  the  appraisal  conduct  and 
results.  The  appraisal  record  is  delivered  to  the  appraisal  sponsor  for  retention. 


•  Collect  and  baseline  appraisal  data  that  becomes  part  of  the  permanent 
records  provided  to  appraisal  stakeholders. 

•  Document  the  satisfaction  of  all  SCAMPI  requirements. 

•  Generate  the  appraisal  record  from  baselined  planning  and  execution  data 
collected  throughout  the  appraisal. 

•  Deliver  the  appraisal  record  to  the  appraisal  sponsor. 


Required  contents  of  the  appraisal  record  include  the  following: 

•  dates  of  the  appraisal 

•  appraisal  input 

•  appraisal  plan 

•  objective  evidence,  or  identification  thereof,  sufficient  to  substantiate 
goal-rating  judgments 

•  characterizations  of  practice  implementation  determined  at  the 
instantiation  level  and  aggregated  at  the  organizational  unit  level 

•  identification  of  the  appraisal  method  (and  version)  used  along  with  any 
tailoring  options 

•  final  findings 

•  all  ratings  rendered  during  the  appraisal  (goals,  PAs,  and  maturity  or 
capability  levels) 

•  ADS 

•  the  set  of  15504  process  profiles  resulting  from  the  appraisal  (if  requested 
by  the  appraisal  sponsor) 

Depending  on  the  recipient  and  intended  usage,  appraisal  data  may  be  subject 
to  being  sanitized  or  edited  in  order  to  comply  with  rules  for  non-attribution, 
confidentiality,  protection  of  proprietary  information,  and  applicable  laws, 
regulations,  or  standards  (e.g.,  acquisition  regulations  or  security 
classification).  Recipients  are  expected  to  place  the  appropriate  limitations  on 
the  access  and  use  of  the  provided  appraisal  data. 

The  appraisal  team  leader  documents  in  the  ADS  that  all  SCAMPI 
requirements  were  satisfied. 


Continued  on  next  page 
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3.2.2  Generate  Appraisal  Record  (continued) 


Optional 

Practices 


Implementation 

Guidance 


Page  11-268 


The  appraisal  record  should  also  contain  any  additional  outputs  requested  by 
the  appraisal  sponsor,  as  agreed  to  during  appraisal  planning  and  documented 
in  the  appraisal  input. 


The  actual  objective  evidence  (artifacts  or  portions  of  artifacts)  need  not  be 
part  of  the  appraisal  record,  but  an  identification  of  the  objective  evidence  is 
required.  This  may  be  implemented  by  providing  the  PIIs  that  were  used  as 
the  basis  for  characterizing  practice  implementation  decisions. 

Guidance  on  the  protection  of  appraisal  data  can  be  summarized  based  on  the 
recipient  of  the  data  as  follows: 

•  appraisal  sponsor:  replacement  of  specific  sources  (persons,  projects)  with  . 
non-attributable,  general  identifiers  (e.g.,  numeric  codes  assigned  to 
projects,  roles,  or  data-gathering  sessions).  If  the  sponsor  is  separate  from 
the  appraised  organization  (e.g.,  in  the  case  of  a  supplier  selection 
context),  there  may  be  situations  where  confidential  or  proprietary  data 
relating  to  the  appraised  organization  must  be  removed. 

•  CMMI  Steward:  same  as  for  appraisal  sponsor,  for  data  that  is  shared  by 
both.  For  data  that  is  provided  only  to  the  CMMI  Steward,  the  data 
collection  vehicles  (e.g.,  forms)  are  already  designed  to  observe  non¬ 
attribution  and  confidentiality  rules.  Additionally,  supplied  data  may  be 
subject  to  further  sanitization  to  comply  with  acquisition  or  security- 
related  restrictions. 

•  senior  site  manager:  in  cases  where  the  appraised  organizational  unit  is 
separate  from  the  appraisal  sponsor,  the  appraised  organization  is 
typically  provided  only  with  appraisal  results  and  not  data  related  to 
planning  and  decision  making,  or  data  that  makes  use  of  the  results. 


♦ 
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3.2.2  Generate  Appraisal  Record 


GSS  and  CPM  Implementation  Guidance 

When  using  the  MDD  for  Government  source  selection  and  contract  process  monitoring,  the 
following  should  be  considered: 

Guidance  for  Regardless  of  the  usage  mode,  security-related  sanitization  is  always  a 
GSS  and  CPM  consideration  when  forwarding  information  to  the  CMMI  Steward. 


When  the  appraisal  is  considered  part  of  the  source  selection,  some  or  all  of 
the  appraisal-related  information  may  be  considered  source-selection 
sensitive.  In  such  cases,  data  supplied  to  the  CMMI  Steward  may  be  subject  to 
further  sanitization  to  comply  with  acquisition  restrictions. 

If  the  appraisal  is  not  considered  part  of  the  source  selection  (for  example,  if 
appraisal  results  have  been  reused  or  the  appraisal  was  completed  before  the 
source  selection  started),  the  results  may  not  be  considered  source-selection 
sensitive.  However,  even  in  these  cases,  the  identification  and/or  numbers  of 
offerors  may  be  source-selection  sensitive  at  least  until  after  contract  award. 
Therefore,  further  sanitization  may  be  necessary,  or  providing  the  results  to 
the  CMMI  Steward  may  have  to  be  delayed. 

Guidance  for  the  ADS  in  GSS  Usage  Mode 

Introduction 

There  are  special  situations  in  which  the  Lead  Appraiser  will  not  be  able  to 
forward  a  complete  ADS  to  the  CMMI  Steward.  Included  in  these  situations 
are  formal  DoD  source  selections  and  appraisals  involving  classified 
organizations,  programs,  or  projects.  The  Federal  Acquisition  Regulations 
(FAR)  cover  legal  requirements  for  conducting  source  selections  and  prohibit 
disclosure  of  the  number  of  offerors  (bidders)  and  their  identities  (e.g.,  FAR 
Part  15.3  and  the  Air  Force  Supplement  to  the  FAR,  AFARS  5315.3). 
Additionally,  national  security  considerations  have  resulted  in  laws 
concerning  the  disclosure  of  classified  information.  In  such  situations,  a 
partially  completed  (i.e.,  sanitized)  ADS  must  be  submitted  to  the  CMMI 
Steward  with  justification  as  to  why  it  is  not  complete. 


Guidance  for 
GSS 


Continued  on  next  page 
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3.2.2  Generate  Appraisal  Record 


GSS  and  CPM  Implementation  Guidance  (continued) 

Guidance  for  Specific  Content  Considerations 

GSS,  cont.  The  following  provides  guidance  concerning  identification  or  itemization  of 
specific  elements  of  the  ADS  in  the  situations  previously  described: 


Appraisal  Disclosure  Element 

GSS  Considerations  for  Reporting 
to  CMMI  Steward 

Appraisal  sponsor  and  sponsor’s 
organizational  affiliation 

Generally  allowed,  unless  the 
organization’s  identity  is  required  to 
be  classified. 

Appraisal  Lead  Appraiser  and 
appraisal  team  members  and  their 
organizational  affiliations 

Generally  allowed,  unless  affiliation 
organizations  are  classified. 

However,  if  the  Lead  Appraiser  or 
appraisal  team  members  are  part  of 
the  source  selection  team,  their 
participation  on  the  source  selection 
is  source-selection  sensitive  (i.e., 
cannot  be  divulged).  Otherwise, 
their  participation  on  the  appraisal 
team  may  be  considered  For  Official 
Use  Only  (FOUO).  Note,  however, 
that  they  can  be  members  of  the 
SCAMPI  appraisal  team  and  not  be 
members  of  the  source  selection 
team  (see  also  the  last  item  below). 

Organizational  unit  appraised  (the 
unit  to  which  the  ratings  are 
applicable  and  the  process  areas 
and/or  domains  examined,  as  defined 
in  the  appraisal  plan) 

For  source  selection  mode,  the 
organization’s  identity  cannot  be 
divulged,  as  well  as  the  identity  of 
programs  or  projects  that  are 
classified.  Unclassified  information 
may  be  divulged  to  the  eventual 
successful  offeror. 

CMMI  model  (version,  representa¬ 
tion,  and  domains) 

Allowed. 

Appraisal  method  (name  and  version) 

Allowed. 

Continued  on  next  page 
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3.2.2  Generate  Appraisal  Record 


GSS  and  CPM  Implementation  Guidance  (continued) 


Guidance  for 
GSS,  cont. 


Appraisal  Disclosure  Element 

GSS  Considerations  for  Reporting 
to  CMMI  Steward 

Process  areas  rated  and  process  areas 
not  rated 

Allowed,  except  as  specified  above. 

Maturity  level  and/or  capability  level 
ratings  assigned 

Allowed,  except  as  specified  above. 

Dates  of  on-site  activity 

Probably  allowed  for  eventual 
successful  offeror,  if  not  classified 
(the  other  dates  would  divulge  the 
number  of  offerors). 

Date  of  issuance  of  the  ADS 

Allowed. 

Statement  affirming  that  all  SCAMPI 
requirements  were  met 

Allowed.  Consider  including  a 
statement  that  all  information  has 
been  divulged  to  the  best  of  the 
Lead  Appraiser’s  ability,  along  with 
justification  as  to  why  some  of  the 
information  could  not  be  divulged. 

Signature  of  appraisal  team  leader  (at 
a  minimum);  those  of  appraisal  team 
members  and  appraisal  sponsor  are 
optional 

Allowed;  however,  the  only  issue  to 
consider  is  whether  they  are 
participating  as  members  of  the 
source  selection  team.  Then  this 
may  be  allowed  after  contract 
award.  If  the  appraisal  team 
members  are  not  part  of  the  source 
selection,  the  appraisal  team,  in 
effect,  passes  the  SCAMPI  results 
over  the  wall  to  the  source  selection 
team  and  participates  no  further.  In 
this  instance,  signatures  would  be 
allowed. 

Guidance  for  Normally  no  additional  sanitization  issues  are  related  to  this  usage  mode. 

CPM 
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3.2.3  Provide  Appraisal  Feedback  to  CMMI  Steward 


Activity  Appraisal  data  required  by  the  CMMI  Steward  is  collected  and  reported.  This 

Description  includes  a  subset  of  the  contents  of  the  appraisal  record,  as  well  other  data 

used  by  the  Steward  to  aggregate  and  analyze  appraisal  performance  data  for 
reporting  to  the  community  and  monitoring  the  quality  of  performed 
appraisals. 

Required  Submit  the  completed  appraisal  report  as  required  by  the  CMMI  Steward. 

Practices 

Parameters  The  CMMI  Steward  defines  the  specific  set  of  data  required  for  submission  at 

and  Limits  the  completion  of  an  appraisal.  Submission  of  the  appraisal  report  is  required 

for  the  appraisal  to  be  recorded  in  the  Steward’s  database  of  appraisal  results. 
This  is  also  a  requirement  established  by  the  Steward  to  maintain  Lead 
Appraiser  authorization. 

Optional  If  the  objective  evidence  is  available  in  electronic  form,  it  can  be  included  as 

Practices  part  of  the  appraisal  report  submitted  to  the  CMMI  Steward. 

Implementation  The  appraisal  team  leader  is  responsible  for  ensuring  that  appraisal  feedback 
Guidance  required  by  the  CMMI  Steward  is  collected  and  reported.  The  CMMI 
Steward,  as  custodian  of  the  product  suite  and  the  Appraiser  Program,  has 
several  objectives  in  seeking  appraisal  feedback: 

•  characterization  of  the  state  of  the  practice  in  the  appraisal  community,  for 
the  collection  and  distribution  of  effective  appraisal  techniques 

•  analysis  of  reported  appraisal  data  to  obtain  an  understanding  of  appraisal 
performance  for  continuous  improvement 

•  quality  control  within  the  Appraiser  Program,  to  ensure  a  high  level  of 
confidence  in  the  accuracy  of  appraisal  results 

Feedback  is  provided  periodically  to  the  community  on  summarized  results 
determined  from  appraisal  data  collected. 

The  format  and  mechanisms  for  the  submission  of  this  data  are  established  by 
the  CMMI  Steward. 
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3.2.3  Provide  Appraisal  Feedback  to  CM  Ml  Steward 


GSS  and  CPM  Implementation  Guidance 

When  using  the  MDD  for  Government  source  selection  and  contract  process  monitoring,  the 
following  should  be  considered: 

Guidance  for  See  guidance  provided  for  activity  3.2.2. 

GSS  and  CPM 

♦ 
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3.2.4  Archive  and/or  Dispose  of  Key  Artifacts 


Activity 

Description 


Required 

Practices 

Parameters 
and  Limits 


Optional 

Practices 

implementation 

Guidance 


After  the  various  reports  are  delivered  to  the  appropriate  stakeholders  and  the 
appraisal  assets  have  been  baselined,  the  team  leader  is  responsible  for 
properly  archiving  and/or  disposing  of  the  appraisal  data,  in  accordance  with 
agreements  made  with  the  sponsor  and  documented  in  the  appraisal  input.  The 
team  librarian  (if  one  is  used)  ensures  that  all  organization-provided 
documentation  and  objective  evidence  is  returned  or  disposed  of  properly. 
Any  remaining  team  artifacts  or  notes  are  disposed  of  properly. 


•  Archive  or  dispose  of  key  artifacts  collected  by  the  appraisal  team. 

•  Return  objective  evidence  provided  by  the  organizational  unit. 


In  all  usage  modes  of  SCAMPI,  strict  non-attribution  policies  apply. 
Confidentiality  and  non-disclosure  agreements  established  with  the  appraisal 
team  members  remain  in  effect. 


None. 


How  the  records  will  be  preserved  or  disposed  of  is  dependent  on  the  usage 
mode  of  the  method  and  the  appraisal  objectives  that  shape  the  current 
application.  Confidentiality  rules  may  differ  by  application.  In  a  supplier 
selection  usage,  the  results  are  not  proprietary  in  that  the  sponsor  is  not  a 
member  of  the  appraised  organization.  However,  results  are  only  known  to  the 
sponsor  and  the  recipient;  competing  organizations  do  not  see  the  results. 
Confidentiality  of  results  can  be  characterized  as  one  of  the  following: 

•  known  only  to  the  recipient  organization 

•  known  to  the  recipient  and  sponsor,  when  they  are  from  different 
organizations 

•  known  to  anyone 

The  sponsor  is  solely  responsible  for  determining  the  confidentiality  with 
which  the  appraisal  results  will  be  maintained.  The  non-attribution  of  data  to 
specific  individuals  is  the  responsibility  of  the  appraisal  team.  The  recipient 
organization,  if  the  sponsor  agrees  and  it  is  planned  for,  may  always  choose  to 
make  the  results  known  outside  the  organization.  At  a  high  level,  this  might  be 
done  for  marketing  and  public  relations  reasons.  Disclosures  of  results  include 
the  context  and  constraints  under  which  the  appraisal  was  performed  (e.g., 
reference  model  scope,  organizational  scope),  as  defined  by  the  ADS 
described  in  process  3.1,  Deliver  Appraisal  Results. 

Any  annotations  related  to  the  objective  evidence  provided  to  the  organization 
by  the  appraisal  team  should  be  recorded  and  archived  for  use  in  process 
improvement  actions  or  for  reuse  in  subsequent  appraisals. 


♦ 
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3.2.4  Archive  and/or  Dispose  of  Key  Artifacts 


GSS  and  CPM  Implementation  Guidance 


No  additional  guidance. 
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Appendix  A  Appraisal  Disclosure 

Statement 


The  Appraisal  Disclosure  Statement  (ADS)  provides  information  considered  essential  to  ade¬ 
quately  interpret  the  meaning  of  maturity  level  or  capability  level  ratings  resulting  from  a 
CMMI  Class  A  appraisal. 

The  ADS  is  prepared  by  the  appraisal  team  leader  and  provided  to  the  appraisal  sponsor  at  the 
conclusion  of  the  appraisal.  If  the  final  findings  briefing  reports  the  appraisal  ratings,  the  ve¬ 
hicle  for  reporting  the  ratings  must  be  the  ADS.  Otherwise  the  appraisal  team  leader  delivers 
the  ADS  to  the  sponsor  as  a  separate  document. 

ADS  Content 

The  ADS  consists  of  the  following  information: 

•  identification  of  appraisal  sponsor  and  sponsor’s  organizational  affiliation 

•  identification  of  appraisal  team  leader  and  appraisal  team  members  and  their  organiza¬ 
tional  affiliations 

•  identification  of  organizational  unit  appraised  (the  unit  to  which  the  ratings  are  applicable 
and  the  domains  examined,  as  defined  in  the  appraisal  plan) 

•  identification  of  CMMI  model  (version,  representation,  and  domains) 

•  identification  of  appraisal  method  (name  and  version) 

•  itemization  of  process  areas  rated  and  process  areas  not  rated 

•  maturity  level  and/or  capability  level  ratings  assigned 

•  dates  of  on-site  activity 

•  date  of  issuance  of  ADS 

•  statement  affirming  that  all  SCAMPI  requirements  were  met 

•  signature  of  appraisal  team  leader  (at  a  minimum);  those  of  appraisal  team  members  and 
appraisal  sponsor  are  optional 
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Appendix  B  The  Role  of  Practice 

Implementation  Indicators  in 
Verifying  Practice 
Implementation 


Purpose 

This  appendix  provides  a  conceptual  overview  of  the  process  of  verifying  practice  imple¬ 
mentation  and  the  role  of  Practice  Implementation  Indicators  in  that  process.  Verification  of 
practice  implementation  is  an  essential  element  of  appraising  the  implementation  of  processes 
relative  to  models  of  best  practices  such  as  CMMI. 

Verifying  Practice  Implementation 

In  this  discussion,  verifying  CMMI  practice  implementation  means  the  substantiation  of 
practice  implementation  based  on  a  review  of  objective  evidence.  For  example,  one  might 
inquire  as  to  whether  a  project-specific  practice  is  implemented  within  a  project.  Alterna¬ 
tively,  one  might  inquire  as  to  whether  an  organization-specific  practice  is  implemented 
within  an  organization. 


Having  a  well-defined  approach  for  verifying  practice  implementation  is  of  critical  impor¬ 
tance  from  several  perspectives.  For  the  process  improvement  sponsor,  it  provides  some  as¬ 
surance  that  the  resources  applied  to  the  improvement  effort  will  result  in  the  desired  out¬ 
come  and  that  the  resultant  benefits  can  therefore  be  expected.  For  process  improvement 
agents  or  champions,  it  enables  them  to  know  when  they  have  succeeded  with  the  imple¬ 
mentation  activity,  and  to  informally  monitor  whether  the  practice  continues  to  be  imple¬ 
mented  over  time.  For  appraisal  teams,  a  well-defined  verification  approach  is  essential  for 
determining  what  capability  level  or  maturity  level  ratings  are  warranted.  CMMI  process  area 
goal  satisfaction  is  predicated  on  implementation  of  the  relevant  specific  or  generic  practices 
(or  acceptable  alternatives)1.  Hence  verification  of  practice  implementation  is  a  crucial  ap¬ 
praisal  task. 


1  See  “Required,  Expected,  and  Informative  Components”  in  Chapter  2  of  the  CMMI  model  that 
you  are  using. 
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Practice  Implementation  Indicators 

The  fundamental  idea  of  Practice  Implementation  Indicators  (PIIs)  is  quite  simple  and 
broadly  applicable  to  any  practice  or  activity.  It  is  based  on  the  obvious  fact  that  the  conduct 
of  an  activity  or  the  implementation  of  a  practice  will  result  in  “footprints” — evidence  that 
the  activity  was  conducted  or  the  practice  was  implemented. 

For  example,  if  one  balances  one’s  checkbook  at  the  end  of  the  month,  there  are  several  po¬ 
tential  ways  to  confirm  that  this  activity  has  indeed  taken  place.  First,  the  person  who  en¬ 
gaged  in  the  checkbook  balancing  activity  can  affirm  that  this  activity  was  conducted.  Sec¬ 
ond,  there  will  likely  be  an  entry  in  the  checkbook  register  for  each  check  or  transaction  to 
indicate  that  it  matches  with  a  corresponding  entry  in  the  bank’s  statement.  Additional  arti¬ 
facts  could  be  identified. 

The  general  idea  is  clear:  the  actual  conduct  of  an  activity  leaves  footprints  that  provide  a 
basis  for  verification. 

PIIs  refer  to  the  footprints  that  are  the  necessary  and  unavoidable  consequence  of  practice 
implementation.  They  include  information  contained  in  artifacts  and  information  gathered 
from  interviews  with  managers  and  practitioners. 

The  Role  of  PIIs 

ARC-compliant  appraisal  methods  employ  objective  evidence  obtained  from  one  or  more 
sources  (instruments,  documents,  and  interviews).  An  appraisal  team  bases  its  decisions  about 
practice  implementation  on  examination  of  this  objective  evidence. 


Once  a  project  or  organization  has  an  understanding  of  how  its  processes  relate  to  the  CMMI 
model,  the  stage  is  set  for  capturing  the  PIIs  that  provide  the  objective  evidence  of  implementa¬ 
tion.  The  work  of  establishing  the  collection  of  PIIs  for  the  project(s)  and/or  organization  pro¬ 
vides  assurance  to  the  process  improvement  sponsor  that  the  expected  implementation  activities 
have  in  fact  resulted  in  alignment  of  the  organization’s  activities  with  the  CMMI  model. 


This  aggregation  of  objective  evidence — the  PIIs — is  itself  an  important  organizational  proc¬ 
ess  asset  that  has  a  number  of  potential  uses,  most  notably  providing  an  appraisal  team  a  head 
start  in  understanding  the  organization’s  implementation  of  the  CMMI  model.  This  leaves  the 
appraisal  team  the  task  of  verifying  whether  the  objective  evidence2  provided  is  adequate  for 
substantiation  of  practice  implementation,  rather  than  the  more  difficult,  error  prone,  and 


2  The  ARC  defines  objective  evidence  as  “qualitative  or  quantitative  information,  records,  or  state* 
ments  of  fact  pertaining  to  the  characteristics  of  an  item  or  service  or  to  the  existence  and  imple¬ 
mentation  of  a  process  element,  which  are  based  on  observation,  measurement,  or  test  and  are 
verifiable.” 
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time-consuming  task  of  investigating  each  practice  to  discover  the  objective  evidence  needed 
to  substantiate  implementation. 


Both  the  appraised  organization  and  the  appraisal  team  have  a  clearer  picture  of  what  artifacts 
need  to  be  provided  to  substantiate  implementation  of  the  practices,  thereby  minimizing  the 
amount  of  further  investigation  necessary  in  the  form  of  interviews  and  additional  documen¬ 
tation  requests.  The  extent  to  which  the  appraised  organization  can  provide  this  information 
becomes  a  principal  factor  in  how  much  further  investigation  may  be  required. 

Another  benefit  of  this  approach  is  significantly  greater  reliability  and  accuracy  of  appraisal. 

The  PH-based  approach  is  not  meant  to  turn  the  appraisal  into  a  documentation  review  exer¬ 
cise.  It  merely  allows  for  more  focused  and  effective  use  of  the  on-site  phase  and  potentially 
a  shorter  on-site  phase  than  would  otherwise  be  the  case. 

Finally,  the  PIIs  are  not  intended  to  tie  the  hands  of  model  implementers  or  process  appraisal 
teams.  The  primary  value  of  the  PIIs  lies  in  making  explicit  what  has  heretofore  been  implicit 
and  therefore  subject  to  wide  variations  in  interpretation  and  understanding.  Over  time,  shar¬ 
ing  of  PIIs  will  result  in  a  set  of  practice  implementation  scenarios  (e.g.,  small,  medium,  and 
large  organizations  or  projects)  and  a  standard  set  of  PIIs  that  could  be  used  as  a  starting 
point  for  further  customization.  The  particular  process  implementation  context  and  the  spe¬ 
cifics  of  the  project  would  determine  which  of  the  indicators  make  sense  for  that  implemen¬ 
tation.  Appraisal  teams  would  be  obliged  to  inquire  into  the  existence  of  the  agreed-upon  in¬ 
dicators,  while  still  having  the  freedom  to  make  judgments  based  on  the  facts  and 
circumstances  of  the  implementation. 

A  standard  set  of  PIIs  could  establish  norms  within  which  most  implementations  will  fall, 
thereby  allowing  efficiencies  to  be  realized  in  implementation  and  appraisal,  while  at  the 
same  time  recognizing  that  alternative  implementations  may  be  possible  using  alternative 
practices. 

Pll  Components 

PIIs  have  two  components  or  dimensions:  an  objective  evidence  component  and  a  practice 
implementation  type  component.  The  objective  evidence  component  refers  to  the  form  of  the 
objective  evidence.  The  practice  implementation  type  component  deals  with  the  significance 
of  the  objective  evidence  in  relation  to  practice  implementation. 
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Forms  of  Objective  Evidence 

An  appraisal  team  bases  its  decisions  about  practice  implementation  on  the  existence  of  ob¬ 
jective  evidence  available  to  it.  This  objective  evidence  can  take  on  one  or  more  of  the  fol¬ 
lowing  forms: 

•  artifacts 

-  work  products,  which  are  the  explicit  intended  consequences  of  practice  implemen¬ 
tation 

-  artifacts  that  are  incidental  to,  but  indicative  of,  practice  implementation 

•  affirmations 

-  written  or  oral  statements  indicative  of  practice  implementation  from  practitioners 
who  carry  out  the  activities  relevant  to  the  practice  or  from  suppliers,  customers,  or 
other  stakeholders  in  the  practice 

-  demonstrations  or  presentations  (e.g.,  the  demonstration  of  capability  of  a  tool  or 
other  mechanism  as  it  relates  to  the  implementation  of  a  practice,  or  a  presentation 
explaining  some  aspect  of  the  organization  or  project) 

Note  that  there  is  not  a  strong  distinction  made  in  the  model  between  artifacts  and  work 
products  (see  Chapter  3  in  the  model  for  an  explanation  of  how  “work  product”  is  used).  As 
used  in  the  context  of  PIIs,  work  product  refers  to  an  artifact  that  is  either  explicitly  men¬ 
tioned  in  the  statement  of  a  CMMI  practice  or  whose  absence  would  be  a  strong  indictor  of 
incomplete  or  inadequate  practice  implementation.  The  weaker  term  “artifact”  is  used  in  the 
context  of  PIIs  to  refer  to  an  artifact  whose  existence  is  incidental  to  (i.e.,  a  side-effect  of)  the 
accomplishment  of  the  main  intent  of  the  practice. 

Types  of  PIIs 

Using  the  above  discussion  as  the  framework,  it  is  now  possible  to  itemize  the  types  of  PIIs 
that  might  be  present  as  a  consequence  of  practice  implementation.  Table  III-l  shows  PII 
types,  which  collectively  provide  coverage  for  any  CMMI  practice.  Each  type  is  described  in 
more  detail  below. 


Table  111-1:  PII  Types 


PII 

Type 

Objective 
Evidence  Form 

Generic  Description 

Direct 

Artifact  (work 
product) 

Work  product(s)  that  reflect  (document  the  information  content  of) 
the  establishment  of  {insert  text  from  practice  statement  that  de¬ 
scribes  object  of  practice  enactment). 

Indirect 

Artifact 

Artifact(s)  that  are  an  indirect  consequence  (or  side-effect)  of  the 
effort  required  to  { insert  text  from  practice  statement  that  describes 
object  of  practice  enactment) . 

Direct 

Affirmation 

Affirmations  from  individuals  who  participated  in  or  contributed  to 
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{insert  text  from  practice  statement  that  describes  object  of  practice 
enactment}  OR  affirmations  from  individuals  who  are  users  of  (or 
who  can  substantiate  use  of)  {insert  text  from  practice  statement 
that  describes  object  of  practice  enactment}. 


Direct  Artifact 

This  PII  type  is  relevant  when  establishment  of  a  work  product  is  an  integral  part  of  practice 
implementation.  Sometimes  this  is  explicit,  as  in  “Establish  and  maintain  process  action  plans 
to  address  improvements  to  the  organization’s  processes  and  related  process  assets”  (OPF  SP 
2.1-1).  In  other  instances,  it  is  not  explicit,  although  it  would  be  difficult  to  imagine  practice 
implementation  without  the  presence  of  one  or  more  work  products  being  produced.  In  most 
cases,  the  model  document  already  identifies  these  work  products. 

Indirect  Artifact 

This  PII  type  applies  to  artifacts  that  are  produced  as  a  natural  consequence  of  practice  en¬ 
actment.  The  difference  between  this  and  a  direct  artifact  PII  is  that  this  type  applies  to  arti¬ 
facts  that  are  an  indirect  consequence  or  side-effect  of  practice  enactment.  For  this  reason, 
artifacts  that  are  relevant  to  this  PII  will  vary  widely  and  will  tend  to  be  implementation- 
specific.  This  indicator  type  is  especially  useful  when  there  may  be  doubts  about  whether  the 
intent  of  the  practice  has  been  met  (e.g.,  a  work  product  exists  but  there  is  no  indication  of 
where  it  came  from  or  who  developed  it). 

Direct  Affirmation 

This  PII  type  refers  to  either  information  obtained  via  interviews  of  individuals  involved  in 
the  enactment  of  a  practice  or  of  individuals  who  are  stakeholders  (e.g.,  customers,  suppliers) 
in  the  enactment  of  a  practice.  This  type  can  also  apply  to  information  provided  in  other 
ways,  such  as  demonstrations  or  presentations. 

PII  Descriptions 

A  PII  Description  (PHD)  is  a  structure  or  schema  defined  to  provide  a  repository  for  the  PII 
information.  Table  III-2  shows  an  example  of  such  a  structure.  Note  that  this  is  a  notional 
description  of  the  content,  not  a  physical  definition  of  the  format. 
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Table  111-2:  A  PHD  Schema 


Attribute 

Synopsis 

Remarks 

Practice  ID 

This  identifies  the  process  area,  goal,  and  practice 
that  the  PII  is  associated  with. 

Acronyms  found  in  the 
CMMI  models  are  used. 

PII  ID 

This  identifies  the  indicator  type  and  the  form  of 
objective  evidence. 

Types  are  direct  artifact, 
indirect  artifact,  and  di¬ 
rect  affirmation. 

Description 

This  is  a  description  of  the  PII  as  applied  to  this 
practice. 

Examples 

These  are  examples  of  artifacts  or  affirmations  that 
would  exemplify  the  intent  of  the  PII  and/or  ex¬ 
ploratory  questions  (EQs)  or  “look  fors”  (LFs).  They 
assist  assessors  in  identifying  relevant  artifacts  or 
eliciting  relevant  information. 

Aim  to  minimize  any 
overlap  with  such  infor¬ 
mation  that  is  already  in 
the  model  document. 

Organizational 

Implementation 

This  attribute  would  be  filled  in  by  the  organization 
as  part  of  its  implementation  program  and  provided 
to  the  appraisal  team  as  a  resource. 

Table  III-3  shows  an  example  PHD  for  specific  practice  1.1-1  of  the  Project  Planning  process 
area: 


Table  111-3:  An  Example  PHD 


Attribute 

Value 

Practice  ID 

PP  SP  1.1-1 

PII  ID 

Direct  Artifact 

PII  Description 

Work  product(s)  that  reflect  (document  the  information  content  of)  the  estab¬ 
lishment  of  a  top-level  work  breakdown  structure  (WBS)  to  estimate  of  the  scope 
of  the  project. 

Examples 

See  Typical  Work  Products. 

Organizational 

Implementation 

{To  be  provided  by  the  organization  for  a  specific  project  implementation.) 

These  descriptions  have  a  number  of  uses  in  addition  to  their  utility  during  process  appraisal 
They  can  be  used  during  the  model  implementation  phase,  after  model  implementation  as  a 
training  vehicle  for  new  personnel,  for  internal  monitoring  of  practice  implementation,  etc. 

Application  of  Plls  in  Model  Implementation 

The  use  of  indicators  has  significant  utility  for  an  organization  that  is  committed  to  model- 
based  process  improvement.  Typically,  organizations  will  either  implement  model  practices 
directly  or  will  ensure  that  the  practices  used  in  the  organization  effect  goal  achievement 
(through  the  mechanism  of  alternative  practices). 
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Since  models  are  necessarily  expressed  and  published  in  an  implementation-independent 
manner,  the  implementation  of  a  model  will  require  that  an  understanding  of  how  the  model 
intent  (as  expressed  though  goals,  practices,  and  other  model  material)  is  to  be  realized  in  the 
organization  be  developed,  documented,  and  operationalized.  The  model  intent  is  made  real 
through  its  impact  on  the  way  people  work;  if  there  is  no  relation  between  how  they  work  and 
the  model,  the  organization  has  not  implemented  the  model.  Thus,  having  an  understanding 
of  the  ways  in  which  implementation  of  the  model  relates  to  what  people  are  doing  in  the  or¬ 
ganization  is  a  necessary  and  unavoidable  prerequisite  to  implementing  the  model.  PIIDs 
provide  a  mechanism  by  which  the  implementation  of  a  model  practice  can  be  described. 

Application  of  Plls  in  Process  Appraisal 

During  the  course  of  process  appraisal,  the  appraisal  team’s  primary  focus  is  on  verifying 
practice  implementation.  This  is  accomplished  by  (1)  obtaining  objective  evidence  relevant  to 
the  implementation  of  a  practice,  (2)  comparing  the  objective  evidence  available  with  what  is 
expected,  and  then  (3)  making  a  determination  of  practice  implementation  based  on  the  dif¬ 
ference  between  actual  and  expected  evidence. 

The  PIIs  assist  the  appraisal  team  (as  well  as  the  implementing  organization)  with  task  1  by 
providing  a  framework  or  structure  that  makes  explicit  the  types  of  objective  evidence  that 
should  be  considered.  In  concert  with  the  CMMI  model  documentation,  this  provides  the 
model  basis  against  which  the  organization’s  actual  operating  practices  are  compared. 


Note  that  PIIs  do  not  prescribe  what  objective  evidence  must  be  present  for  practice  imple¬ 
mentation  determinations  to  be  made;  they  only  make  explicit  what  is  reasonable  for  an  ap¬ 
praisal  team  to  consider.  The  particular  circumstances  and  attributes  of  the  organizational  unit 
and/or  project  must  all  be  taken  into  consideration  when  making  determinations  of  practice 
implementation.  As  a  general  rule,  the  more  objective  evidence  and  the  more  PIIs  represented 
by  that  objective  evidence,  the  higher  the  confidence  level  that  the  practice  is  implemented. 


The  PII  structure  assists  the  appraisal  team  with  task  2  to  the  extent  that  the  team  has  agreed 
in  advance  on  the  objective  evidence  it  expects  to  see  for  each  process  instantiation  exam¬ 
ined.  In  some  cases  it  may  be  difficult  or  impossible  to  have  completely  developed  a  team 
consensus  on  what  objective  evidence  must  be  seen  (in  advance).  But  sooner  or  later  the  ap¬ 
praisal  team  must  establish  a  consensus  view  on  what  is  reasonable  to  expect,  since  it  is  only 
the  presence  of  that  consensus  view  that  permits  a  determination  of  practice  implementation 
to  be  made. 

The  final  practice  implementation  determination  task  is  that  of  developing  a  team  consensus 
on  whether  the  practice  is  implemented  for  the  process  instantiation  being  examined.  This 
decision  is  based  on  the  difference  between  what  is  expected  and  what  is  observed. 
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Appendix  C  Focused  Investigation 

Elaboration  and  Guidance 


Concept  Description 

This  appendix  describes  the  use  of  preliminary  objective  evidence  review,  continuous  con¬ 
solidation  of  objective  evidence,  and  practice  characterization  in  focusing  the  data  collection 
and  investigation  effort  of  the  appraisal  team. 

Focused  investigation  relies  on  a  high  degree  of  planning,  organization,  and  subsequent  man¬ 
agement  and  control  of  the  activities  of  the  appraisal.  The  concept  incorporates  the  following 
activities: 

•  collecting  preliminary  objective  evidence  through  instruments  as  a  part  of  obtaining  pre¬ 
liminary  data 

•  creating  an  inventory  of  objective  evidence  collected,  to  support  practice  implementation 
characterization 

•  initially  reviewing  and  analyzing  preliminary  objective  evidence  inventoried,  to  identify 
gaps  in  objective  evidence  supporting  practice  characterization 

•  identifying  information  needs  to  support  initial  preparation  and  refinement  of  the  data 
collection  plan 

•  continuously  consolidating  objective  evidence  collected  and  updating  the  status  of  prac¬ 
tice  characterization  for  each  organizational  unit  instantiation  (aggregated  up  to  the  orga¬ 
nizational  unit) 

Preliminary  Focused  Investigation 

Focused  investigation  should  be  begun  in  the  Appraisal  Planning  phase  of  the  appraisal.  Fo¬ 
cused  investigation  is  best  initiated  with  a  practice-based  initial  data  collection  instrument 
that  documents  the  organizational  unit’s  implementation  of  the  practices  of  the  CMMI  model 
for  each  instantiation  within  the  scope  of  the  appraisal.  Preliminary  data  may  be  collected 
using  instruments  such  as  questionnaires,  surveys,  and  presentations.  This  data  should  in¬ 
clude  a  preliminary  inventory  of  Practice  Implementation  Indicators. 


An  inventory  and  review  of  this  data  provides  an  important  initial  determination  of  the  gaps 
in  the  data  available  supporting  practice  implementation,  as  well  as  what  information  and 
objective  evidence  is  needed.  These  activities  are  performed  as  part  of  the  Obtain  and  Ana- 
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lyze  Preliminary  Objective  Evidence  process.  The  more  complete  and  comprehensive  this 
early  data  collection  and  analysis  is,  the  better  prepared  the  organizational  unit  will  be  for  the 
appraisal. 

These  preliminary  information  needs  can  provide  the  foundation  of  the  data  collection  plan 
for  the  remainder  of  the  appraisal  process.  They  also  provide  the  foundation  for  the  readiness 
review  and  any  necessary  adjustments  in  the  appraisal  plan,  providing  a  clearer  set  of  initial 
expectations  for  the  magnitude  of  the  appraisal  effort.  Data  gaps  found  can  result  in  addi¬ 
tional  document  requests  and  other  data  collection  plans.  These  activities  are  performed  as  an 
early  part  of  the  Prepare  for  Collection  of  Objective  Evidence  process. 

Continuous  Consolidation  and  Tracking 

Following  the  initial  focused  investigation  effort,  a  data  collection  plan  is  developed  and  fol¬ 
lowed.  Data  collection  activities  are  described  by  the  Examine  Objective  Evidence  process. 
This  process  typically  consists  of  planned  data  collection  activities  that  include  presentations, 
document  reviews,  and  interviews. 

As  these  data  collection  activities  are  performed,  practice  characterization  and  strengths  and 
weaknesses  are  recorded  and  added  to  the  existing  objective  evidence  inventory,  and  continu¬ 
ously  reviewed  (see  the  Verily  and  Validate  Objective  Evidence  and  Document  Objective 
Evidence  processes).  Additional  data  collected  is  added  and  consolidated  with  the  data  al¬ 
ready  collected  to  continuously  provide  the  assessment  team  with  a  view  of  their  progress 
against  the  data  collection  plan  and  model  coverage.  This  is  referred  to  as  “continuous  con¬ 
solidation.” 

Monitoring  and  controlling  the  data  collection  plan  and  model  coverage  is  an  essential  aspect  of 
performing  focused  investigation  and  continuous  consolidation.  The  appraisal  team  must  be 
able  to  record,  monitor,  and  track  progress  against  the  data  collection  plan.  This  may  be  done  in 
several  ways,  but  generally  requires  the  use  of  some  mechanism  for  recording  the  progress  to¬ 
wards  determining  practice  characterization  for  each  reference  model  practice  within  the  scope 
of  the  appraisal.  As  data  is  collected  for  each  practice,  and  for  each  sample  instantiation  of  the 
organizational  unit  being  appraised,  it  is  also  useful  to  have  some  mechanism  for  easily  com¬ 
paring  and  consolidating  practice  implementation.  Instruments  and  automated  tools  that  support 
the  Conduct  Appraisal  phase  of  the  appraisal  can  greatly  facilitate  this. 


Perhaps  the  most  important  feature  of  focused  investigation  is  the  appraisal  team’s  awareness 
of  its  status  regarding  determination  of  practice  characterization  and  goal  satisfaction.  The 
team  continually  maintains  an  understanding  of  how  the  data  collected  supports  the  imple¬ 
mentation  of  each  practice  for  each  instantiation  of  the  organizational  unit,  and  what  addi¬ 
tional  objective  evidence  is  needed.  This  allows  the  team  to  update  the  data  collection  plan  to 
optimally  refocus  their  efforts  during  the  data  collection  activities. 


111-14 


CMU/SEI-2002-HB-002 


Appendix  D  ARC/MDD  Traceability  Table 
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ARC  ID _ ARC  Requirement _ MOD  Process _ MDD  Activity _ Notes/Comments _ 

X.  1 .2  The  method  shall  define  the  responsibilities  of  the  appraisal  1 .3  Select  and  Prepare  1.3.1  Identify  Team  Leader  Appraisal  team  leader  responsibilities  are 
team  leader,  which  at  a  minimum  shall  include  the  following  Team  defined  throughout,  but  are  summarized  in 

activities:  1.3.1. 


111-16  CM  U/SEI-2002-HB-002 


CM  U/S  E 1-2002-H  B-002  111-17 


111-18  CMU/SEI-2002-HB-002 


- 

s  s 

M 

c  . 
■X3  >»  -£2 

c 

M  C  O 

o  !J  | 

Ti  g  “ 

8  •§  S 


to  £ 

3  -o 

1  § 

P  C/5 

ill 
1  !  i 

o  (D 
s  5  ^ 

c  5  g 

p  §><2 

l|  8 

o  s  ^ 
o  J3  C 
T3  o  .2 

"§•“  g. 
£  &  &* 
g  c  ? 
S  a  _g 
«  S  £ 

jC  «3  O 
£.  g»  9 

u  -5  -| 
5  2  « 

<  In  ID 


O 

'+-'  bi)  V5 

||1 
S  3  a- 
TJ  G  o 

1-Sl 

■8  §|" 
'>  S  2 

2  4  2 

£"3  & 

«  .3  S 

■f 

5  9<tj 

O  C3  (U 
2  °| 
i^'s 
II  § 

1 1 « 
^  T3  d 
^  tfl  T3 
g  C<H 
-Coo 
to  °  c 

6  ^  .2 
to  ^  B 
-C 

£-S  •£ 

5  S  § 


.2.14 

.2.15 

_JSt _ 

U3 _ 1 

CM  U/SEI-2002-H  B-002  111-19 


©  73 

V  -B  .g 

«  S  2 

Oh 

S  N  §• 

Z  V 

cp  *£ 

o  o 


tr  w.2 

£  -c  c 

„  a  « 

0*^2 

SUo 

5Sfi 

IS  o 

§>=s 

t>  R 

T3  °  W5 

1  I  « 

I73  | 

a*  u  u 

3-s| 

uc  c  t3 

«  D  e 

-  (L>  5 
S  >  & 

2  £  tfl 

g  t>  M 

2  pD  fi 

c  ^  K3 

».ge 

£  35  w 
g  •£  <u 

o  S  *5 

2  §  -2 
TJ  fa  § 

JS  bD  ^ 

III 

y  o  a 

e  g  s 


w  o  . 


•£  .2  ■= 

on  o 

73  *  e 

3  S  O 
O  CL  £< 

c  e  05 

-r*  D 

^  73  £ 

8  1^ 

^  a  o 

5  «  -o 

'SJJ  g 

S-<T  & 

cu  3?  cs 
C3  o 
o  c  A* 
-C  o  13 


33 

o 

c 

4-1 

O 

<2J 

on 

<5 

<u 

3 

■5 

3 

'H 

■> 

c 

0) 

</: 

Up 

o 

3 

Cu 

<L) 

o 

c 

73 

3 

<U 

•5 

4- 

O 

T3 

a)  73  a 

•c 

2  i-i 

o 

O  D,  U 

on 

<u 

C  b  JD 

T3 

05 

_^_ 

O  o  a  ' 

a  3  ’5 

-U  Vt  -VI, 

> 

> 

> 

> 

£ 

JC 

X 

1c 

o 

o 

o 

o 

Up 

Up 

Up 

Up 

< 

< 

< 

< 

*o 

on 

T3 

on 

T3 

on 

T>  w 

c 

15 

3 

t> 

3 

« 

3  <y 

3 

on 

on 

3 

on 

on 

3 

on 

on 

^  03 

<L> 

bo 

< 

<D 

bo 

< 

<U 

bo 

< 

a< 

3 

(J 

73 

,w 

3 

O 

73 

on 

3 

o 

73 

on 

3  -32 

o  .2 

3 

’S3 

3 

*s 

3 

3 

3  3 

di 

Dp 

Cu 

p , 

Cu 

Dp 

Cu  fc. 

<N 

Dp 

pj 

& 

<N 

Dp 

<N  CU 

CP 

< 

CP 

< 

rP 

< 

CP  < 

1  s  :g 

c  a  c 

T3  (D  3 

§~  bO 

73  c 

bJG  .«  ’> 

s  5  > 

■aa  j-  o 
as 
&  &,o 

g  3  <+h 

o  <L>  <L> 

O  £  -g 

Sfi.s 

■0  9  5 

§  |i 

on  «  w 

'H  «  J 

8  -s  - 

P  T3  T3 
*■*  a  3 

^  o  a 

111 

Cl,  a  « 

&.a  2 

«  a  ^ 
o  g-g  i; 

a»  9,l  i 
■|i  2  1 

8  w  g>  8 

p«  .5  .£  c 

J>  c  TJ  C 

‘a  -i  e  | 


a 

3 

pD 

u 

\D 

VD 

H 

— H 

— p 

< 

Pi 

pi 

r+ 

pi 

PT+ 

HI-20  CM  U/SE 1-2002-H  B-002 


ARC  ID _ ARC  Requirement _ MDD  Process _ MOD  Activity _ Notes/Comments 

1-.3.1  .a  (ABC)  the  purpose  of  the  appraisal  1 .4  Obtain  and  Analyze  1 .4. 1  Prepare  Participants 
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ARC  ID _ ARC  Requirement _ MDD  Process _ MDD  Activity _ Notes/Comments 

1.5.7  (A)  The  method  shall  require  that  the  appraisal  participants  be  2.2  Verify  and  Validate  3.7.3  Validate  Practice  Validation  of  preliminary  findings  is  ad 

presented  with  the  draft  findings  in  order  to  solicit  their  re-  Objective  Evidence  Implementation  Gaps  dressed  in  3.7.3. 

sponses  for  verification  of  the  findings’  accuracy  and  clarity. 
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Appendix  E  Government  Source 

Selection  and  Contract 
Process  Monitoring  Context 
and  Considerations 


Background 

Guidance  for  the  use  of  appraisal  methods  utilizing  the  Capability  Maturity  Model®  (CMM®) 
framework  in  acquisitions  has  its  roots  in  the  Department  of  Defense  Directive  (DoDD)  5000 
series  and  in  Office  of  the  Secretary  of  Defense  (OSD)  policy.  (A  copy  of  the  policy  memo¬ 
randum  for  software  evaluations  is  available  in  Appendix  G,.)  The  Software  Capability 
Evaluation  (SCEsm)  method  based  on  the  Capability  Maturity  Model  for  Software  (SW- 
CMM)  and  Software  Development  Capability  Evaluations  (SDCEs)  has  been  routinely  used 
in  source  selection  and  contract  process  monitoring  activities  for  a  number  of  years.  Com¬ 
mercial  industry  began  using  SCEs  for  analogous  subcontractor  selection  and  monitoring 
more  recently  with  analogous  experience  and  results. 

Software  Capability  Evaluations  (Legacy  Experience) 

Traditionally,  capability  evaluations  have  helped  acquisition  managers  achieve  the  following 
goals: 

•  Identify  risks  by  evaluating  process  capability  in  supplier  selection. 

•  Manage  risk  by  motivating  contractors  to  improve  their  development  processes,  without 
forcing  compliance  to  specific  practices. 

•  Monitor  award  fee  incentives  for  contractors  who  have  structured  process  improvement 
programs. 


The  use  of  “external  capability  evaluations”  as  a  means  of  independent  validation  of  organi¬ 
zations’  development  process  maturity  and  capability  by  commercial  and  Government  orga¬ 
nizations  with  oversight  responsibilities  (e.g.,  Government  Accounting  Office  and  Defense 
Contract  Management  Agency)  has  been  steady  and  routine. 


®  Capability  Maturity  Model  and  CMM  are  registered  in  the  U.S.  Patent  and  Trademark  Office  by 
Carnegie  Mellon  University. 

SM  SCE  is  a  service  mark  of  Carnegie  Mellon  University. 
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SCAMPI 


CMMI  and  its  associated  appraisal  method,  SCAMPI,  are  slated  to  replace  the  SW-CMM  and 
its  associated  appraisal  methodologies:  CMM-Based  Appraisal  for  Internal  Process  Im¬ 
provement  (CBA-IPI),  VI. 2  and  Software  Capability  Evaluation  (SCE),  V3.0.  SCAMPI  is 
expected  to  be  the  single  appraisal  methodology  to  be  appropriately  tailored  for  use  with 
CMMI  models. 


SCAMPI  fulfills  the  Appraisal  Requirements  for  CMMI  (ARC)  VI.  1  document  that  required 
an  appraisal  methodology  that  can  be  used  (a)  to  benchmark  process  improvement  efforts 
internal  to  contractor  organizations  and  ( b )  for  Government  and  commercial  supplier- 
selection  and  contract-monitoring  applications.  As  a  benchmarking  appraisal  method, 
SCAMPI  is  classified  as  an  ARC  Class  A  method.  ARC  Class  B  and  C  methods  have  less 
stringent  appraisal  requirements. 


With  the  advent  of  SCAMPI  V  1.1,  the  historical  use  of  the  terms  “assessment"  and  “evalua¬ 
tion"  are  replaced  by  the  term  “appraisal."  What  were  formerly  assessments  and  evaluations 
will  instead  use  the  same  core  appraisal  methodology  with  tailoring  guidance  appropriate  to 
the  circumstances  of  internal  process  improvement,  supplier  selection,  or  contract  process 
monitoring.  Although  the  ARC  defines  three  classes  of  appraisals  (e.g.,  A,  B,  and  C),  it  is  be¬ 
yond  the  scope  of  this  appendix  to  address  appraisal  classes  other  than  the  benchmarking 
methodology  of  SCAMPI  V 1 . 1  at  this  time. 

Of  note  is  the  work  of  the  Software  Evaluation  Integrated  Product  Team  (IPT)  sponsored  by 
the  Deputy  Under  Secretary  of  Defense  for  Science  and  Technology  and  chartered  in  2001. 
Their  document,  “Requirements  for  Process  Evaluation  Methods  and  Their  Application," 
provided  significant  input  to  the  SCAMPI  Assessment  Methods  Integration  Team  and  the 
development  of  the  SCAMPI  method. 

Overview  of  SCAMPI  for  Government  Source 
Selection  and  Contract  Process  Monitoring 

SCAMPI  typically  will  be  used  in  two  different  acquisition  environments:  Government 
source  selection  (supplier  selection)  and  contract  process  monitoring.  Supplier  source  selec¬ 
tion,  the  application  for  which  SCE  was  originally  developed  and  which  SCAMPI  will  re¬ 
place,  has  been  in  routine  use  since  the  original  publication  of  the  CMM  concepts.  Current 
trends,  however,  have  seen  a  consistent  application  of  SCEs  in  the  post-contract  award  envi¬ 
ronment.  Similarly,  the  commercial  community  has  been  applying  SCEs  in  the  selection  of 
subcontractors  and  teaming  partners.  It  is  expected  that  the  demand  in  these  environments 
will  continue,  but  these  demands  will  be  satisfied  by  the  application  of  SCAMPI  instead  of 
SCEs. 
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Factors  to  consider  before  using  SCAMPI  in  an  acquisition  include  the  following: 

•  How  critical  is  the  component? 

•  Do  you  lack  data  about  the  offeror’s  past  performance  or  product  development  capabil¬ 
ity? 

•  What  is  the  total  dollar  value  of  the  acquisition  or  component? 

•  What  is  the  priority  of  management  control  in  this  acquisition? 

•  Are  the  mission  needs  unprecedented? 

•  What  is  the  current  acquisition  life-cycle  phase? 

•  What  is  the  length  of  time  needed  for  the  acquisition? 

•  What  is  the  size  of  the  acquired  component,  including  the  number  of  configuration  items? 

•  How  good  is  the  relationship  between  the  prime  contractor  and  subcontractor? 

SCAMPI  in  Source  Selection 

The  factors  listed  above  affect  the  implementation  of  SCAMPI  and  become  visible  in  the 
following  acquisition  documentation: 

•  Federal  Business  Opportunities  (FBO)/Commerce  Business  Daily  (CBD)  or  similar  an¬ 
nouncement 

•  source  selection  plan  (SSP) 

•  evaluation  plan  (EP) 

•  bidder’s  briefing 

•  request  for  proposal  (RFP) 

•  statement  of  objectives 

•  award  fee  plan  (possibly) 

•  briefing  to  successful  offeror 

•  briefing  to  unsuccessful  offerors 

When  used  effectively,  virtually  every  major  activity  in  a  source  selection  is  affected  by 
SCAMPI.  Each  of  these  documents,  particularly  the  SSP,  EP,  and  RFP,  facilitates  the  use  of 
SCAMPI  during  the  source  selection  evaluation.  Figure  III-l  illustrates  a  global  view  of  a 
notional  source  selection  schedule  that  includes  SCAMPI  activities. 
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Phase  I  Phase  II  Phase  III 


Decision  to  use 

RFP  is 
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■ 
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« 

• 

• 

The  SSEB  The  SSAC  performs  a  The  SSA  makes  the 

evaluates  the  risk/benefit  assessment  award  decision 

offerors 


For  each  offeror: 

•  Determine,  prioritize,  and  finalize  the  data  collection  plan  (1*5  days). 

•  Visit  the  site  (5-10  days). 

•  Produce  a  findings  report  (2-3  days). 


Figure  111-1:  SCAMPI  Activities  in  a  Notional  Source  Selection  Timeline 


The  following  paragraphs  describe  each  phase  of  the  timeline  using  SCAMPI  for  supplier 
selection. 

Phase  I  -  Decision  Point  to  RFP  Release 

The  decision  to  use  SCAMPI  immediately  sets  things  in  motion  for  appraisal  planning  and 
implementation.  Nominally,  the  decision  is  articulated  in  the  source  selection  plan,  and  de¬ 
tailed  usage  of  the  determination  of  SCAMPI  evaluation  results  is  delineated  in  the  source 
selection  evaluation  plan.  Appropriate  language  is  selected  and  tailored  for  insertion  in  the 
request  for  proposal  requiring  SCAMPI  usage  and  how  the  offerors  are  to  provide  SCAMPI- 
related  information  to  the  source  selection  team.  Selecting  the  SCAMPI  Lead  AppraiserSM 
and  SCAMPI  team  members  and  training  them  will  normally  occur  prior  to  proposal  receipt 
because  the  SCAMPI  team  is  not  necessarily  in  place  prior  to  RFP  release.  However,  a 
SCAMPI-knowledgeable  person  is  needed  to  participate  in  the  planning  and  preparation  of 
the  RFP  for  the  SCAMPI  evaluation. 


SM  SCAMPI  Lead  Appraiser  is  a  service  mark  of  Carnegie  Mellon  University. 
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Phase  II  -  RFP  Release  to  Proposal  Receipt 


This  phase  of  the  supplier  selection  timeline  is  an  opportunity  to  bring  the  SCAMPI  team 
together  (if  not  already  done),  provide  SCAMPI  team  training,  and  familiarize  the  team  with 
program  requirements  and  risk  areas. 

Phase  III  -  Proposal  Receipt  to  Site  Visit 

Following  proposal  receipt,  the  appraisal  team  determines  the  specific  data  collection  plan  to 
be  carried  out  for  each  offeror  determined  to  be  responsive  to  the  source  selection  require¬ 
ments.  An  appraisal  plan  defines  the  organizational  scope  and  the  CMMI  model  scope,  which 
are  the  precursors  for  defining  the  explicit  data  collection  strategy.  Definition  of  these  factors 
includes  specific  determination  of  which  CMMI  representation  (staged  or  continuous)  would 
be  used,  the  targeted  maturity  or  capability  levels,  and  the  process  areas  (PAs)  that  will  be 
evaluated.  During  the  on-site  period,  the  team  collects  information  and  turns  the  information 
into  findings  in  the  form  of  strengths,  weaknesses,  and  improvement  activities  (if  requested 
by  the  sponsor).  At  a  minimum,  ratings  of  PA  goals  based  on  the  aggregate  of  strengths  and 
weaknesses  of  the  organizational  unit  within  the  scope  of  the  appraisal  will  be  determined 
and  reported  as  part  of  the  appraisal  disclosure  statement  (ADS).  (See  the  SCAMPI  Vl.l 
Method  Definition  Document  Appendix  A  for  a  description  of  an  ADS.)  The  data  and  find¬ 
ings  are  then  provided  to  the  sponsoring  organization  in  the  format  agreed  on. 

In  most  source  selections,  the  SCAMPI  team  is  one  of  a  number  of  teams  involved  in  pro¬ 
viding  evaluation  services  to  the  Source  Selection  Evaluation  Board  (SSEB).  Typically,  there 
are  other  teams  evaluating  criteria  in  management,  cost,  and  other  technical  areas.  These 
teams  provide  their  findings — just  as  the  SCAMPI  team  provides  their  findings — according 
to  the  SSP. 


For  example,  the  SSEB  evaluates  development  organizations’  proposals  for  an  acquisition 
relative  to  a  prescribed/published  set  of  proposal  evaluation  criteria  and  identifies  the  risks 
(relative  to  the  evaluation  criteria)  of  development  organizations  being  able  to  fully  execute  a 
contract  if  awarded  to  them.  This  risk  assessment  (relative  to  the  proposal  evaluation  criteria) 
is  provided  to  the  Source  Selection  Advisory  Council/Committee  (SSAC). 

The  SSAC’s  responsibility  is  that  of  overall  risk  assessment  of  suppliers  determined  to  be 
responsive.  The  SSAC  assesses  the  overall  risks  of  selecting  each  olferor  and  provides  its 
assessment  of  risk  to  the  Source  Selection  Authority  (SSA),  which  is  empowered  to  make  an 
award  of  an  executable  contract.  The  SSA’s  responsibility  is  to  make  an  award  decision  that 
minimizes  risks  and  maximizes  the  benefits  to  the  sponsor. 

SCAMPI  in  Contract  Process  Monitoring 

The  value  of  implementing  SCAMPI  in  source  selection  can  continue  past  the  contract  award 
and  into  contract  performance.  The  source-selection  SCAMPI  identifies  a  set  of  risks  associ- 
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ated  with  the  successful  offeror.  Those  same  risks,  defined  as  weaknesses  associated  with 
individual  process  areas,  can  be  tracked  or  monitored  as  the  contract  progresses  if  the  pro¬ 
gram  office  feels  that  improvement  in  those  areas  will  benefit  program  development.  Im¬ 
provements  can  be  monitored  by  doing  the  following: 

•  using  weaknesses  to  define  the  risks 

•  developing  a  plan  to  mitigate  the  risks 

•  performing  trade-off  analysis  to  establish  levels  of  surveillance  for  strong  or  weak  areas 

•  defining  the  adequate  reporting  or  insight  to  be  provided  to  the  program  office  to  facili¬ 
tate  monitoring 


In  contemplating  using  SCAMPI  as  a  contract  process  monitoring,  risk-management  tool,  the 
following  questions  could  be  considered: 

•  What  would  you  like  (and  need)  to  know  at  the  start  of  the  contract? 

•  What  expertise  would  the  program  office  need  to  monitor  performance? 

•  What  action  should  be  taken  at  the  start  of  the  contract? 

•  What  action  should  be  taken  if  identified  risks  occur? 

•  Should  there  be  incentives  to  motivate  mitigation  of  the  identified  risks,  possibly  through 
vehicles  such  as  the  program  award  fee? 

•  What  are  the  major  risks  that  could  be  mitigated  by  the  supplier  with  a  process  improve¬ 
ment  effort? 

Use  SCAMPI  appraisal  findings  to  define  the  risks  associated  with  the  execution  of  the  con¬ 
tract,  to  develop  a  plan  to  mitigate  those  risks,  and  to  work  the  plan.  This  plan  could  include 
such  items  as  trading  off  the  surveillance  of  strong  areas  for  weak  ones.  If  an  organization  is 
found  to  have  excellent  configuration  management  procedures,  it  is  wasteful  to  check  on  this 
process  area  in  the  same  way  that  would  be  applied  to  an  area  found  to  be  weak  (e.g.,  Project 
Monitoring  and  Control). 

Using  SCAMPI  to  Baseline  Performance 

As  has  been  done  historically  with  SCEs,  SCAMPI  can  be  used  to  establish  baseline  contract 
process  performance.  One  strategy  that  could  be  used  is  to  establish  a  baseline  of  the  devel¬ 
opment  organization’s  performance  relative  to  a  CMMI  model.  The  creation  of  this  baseline 
entails  a  number  of  planning  and  execution  factors. 

For  New  Contracts 

The  salient  points  to  be  integrated  into  a  plan  for  use  of  SCAMPI  for  new  contracts  are  as 
follows: 

•  The  RFP  must  identify  SCAMPI  for  use  in  contract  process  monitoring  (i.e.,  perform  a 
SCAMPI  appraisal  in  source  selection  and  then  use  SCAMPI  to  monitor  the  contract). 
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•  SCAMPI  is  still  an  evaluation  factor  in  selection. 

•  Contract  process  monitoring  criteria  should  be  based  on  program  office  needs  and  be 
identified  in  the  contract  or  statement  of  work  (SOW).  The  following  issues  should  be  as¬ 
sessed  for  inclusion  in  the  contract  or  statement  of  work: 

-  mitigating  weaknesses 

-  creating  additional  strengths 

-  improving  actual  versus  planned  tracking  of  improvement  activities 
within  process  areas 

A  third  Government  acquisition  environment,  separate,  but  inclusive  of  most  of  the  factors 
involved  with  source  selection,  is  that  of  “sole  source  procurements.”  In  this  environment, 
whereby  the  Government  will  be  awarding  an  individual  organization  a  contract  without 
competition,  SCAMPI  is  not  used  as  a  selection  discriminator  but  as  a  means  to  provide  the 
Government  with  contractor  process  maturity  data  very  early  in  the  sole  source  program. 
Identifying  process  weaknesses  prior  to  contract  award  can  be  used  to  negotiate  a  required 
risk  mitigation  plan  to  address  contractor  process  weaknesses.  The  points  listed  above  in  the 
previous  paragraph  apply  in  a  similar  fashion  when  considering  the  planning  for  a  sole  source 
procurement.  The  most  significant  factor  surrounding  the  use  of  SCAMPI  or  any  appraisal  in 
this  environment  is  the  ability  of  the  acquisition  agency  to  incorporate  the  requirements  and 
results  into  the  procurement  or  acquisition  plans. 

For  Existing  Contracts 

The  points  to  be  integrated  into  a  plan  for  using  SCAMPI  for  existing  contracts  when 
SCAMPI  can  be  used  as  a  contract  process  monitoring  tool  are  as  follows: 

•  Using  SCAMPI  for  contract  process  monitoring  can  be  a  negotiated  change  to  the  con¬ 
tract. 

•  When  a  long-term  relationship  is  expected  and  the  benefits  of  process  improvement  can 
be  realized,  SCAMPI  is  a  good  choice  for  contract  process  monitoring. 

•  Refer  to  the  same  criteria  as  for  new  contracts  (above). 


Award  Fees 

Establishing  a  process  baseline  lends  further  utility  to  SCAMPI  in  contract  process  monitor¬ 
ing  for  considering  award  fees  or  value  engineering  incentives  for  process  improvement. 
Note,  however,  that  award  fee  applications  (e.g.,  an  award  for  meeting  specified  measures  of 
performance)  are  not  appropriate  in  all  instances.  The  award  fee  application  of  SCAMPI  is 
most  appropriate  under  the  following  circumstances: 

•  A  long-term  relationship  is  involved. 
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•  The  contractor  lacks  a  sufficient  number  of  programs  over  which  to  spread  improvement 
costs. 

•  Process  investments  in  general  would  not  otherwise  be  made. 

•  The  sponsoring  organization  believes  direct  investment  incentives  will  be  the  best  moti¬ 
vator  of  action. 

•  The  program  environment  includes 

-  mission-critical  systems 

-  embedded  systems 

-  a  history  of  system/software  engineering  issues 

•  SCAMPI  is  used  by  the  sponsoring  organization  to  mitigate  risks. 

•  The  objective  and  ultimate  goal  of  applying  SCAMPI  are  the  following: 

-  Objective:  Provide  incentive  for  contractors  to  improve  the  total  systems  engineering 
and  software  development  process. 

-  Goal:  Exceed  the  product  development  quality,  cost,  and  schedule  requirements. 

A  Teaming  Approach 

The  sponsoring  organization  and  contractor  should  view  themselves  as  team  members  in  an 
effort  to  benefit  from  an  overall  systems  engineering  and  software  engineering  process  im¬ 
provement  plan.  This  teaming  approach  has  some  specific  characteristics: 

•  CMMI  is  the  basis  for  the  improvement  effort. 

-  The  contractor  uses  CMMI  to  establish  plans. 

-  The  sponsoring  organization  evaluates  its  processes  using  CMMI. 

•  Contract  incentive  is  the  contractual  vehicle. 

-  The  contract  describes  the  sponsoring  organization’s  goals. 

-  The  contract  describes  the  method  of  evaluating  progress. 

•  The  sponsoring  organization  and  contractor  jointly  agree  to  the  criteria  and  approach. 

•  The  award  fee  plan  increments  and  criteria  support  long-range  objectives. 

-  The  award  fee  plan  increments  and  criteria  can  be  tailored  specifically  to  program 
needs. 

-  SCAMPI  is  used  to  establish  baseline  systems/software  engineering  process  capabil¬ 
ity. 

-  Findings  are  provided  to  the  contractor. 

-  The  contractor  uses  findings  to  focus  the  improvement  plan. 

-  The  sponsoring  organization  and  contractor  jointly  agree  to  goals. 

-  SCAMPI  is  then  used  to  measure  progress  against  the  improvement  plan. 

-  Incentive  awards  are  determined  by  the  contract  provisions. 
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Findings  establish  the  new  baseline  for  the  next  increment. 


The  keys  to  the  successful  application  of  award  fee  usage  of  SCAMPI  are  to  (a)  perform  the 
source-selection  SCAMPI  evaluation,  ( b )  use  the  findings  to  frame  the  award  fee  plan,  (c) 
perform  a  baseline  SCAMPI  evaluation  (after  a  suitable  time  frame  [six  months]  that  allows 
the  contractor  to  begin  contract  performance),  ( d )  have  the  contractor  submit  a  process  im¬ 
provement  plan  (PIP),  and  (e)  involve  the  contractor  to  obtain  an  understanding  of  the 
SCAMPI  findings  and  impacts  upon  the  award  fee  pool. 

Value  Engineering 

Value  engineering  for  product  development  process  improvement  is  another  mechanism  that 
is  available.  Value  engineering  is  described  in  the  Federal  Acquisition  Regulations  (FAR)  Part 
48  and  is  extensible  to  process  improvement.  There  are  five  elements  required: 

1.  FAR  clause  52.248-1 

2.  separately  identifiable  systems/software  engineering  work  packages  in  an  earned  value 
system 

3.  a  baseline  of  prices  for  the  systems/software  engineering  effort 

4.  SCAMPI  (to  establish  a  process  baseline  and  validate  process  improvements) 

5.  a  SOW  requirement  (to  develop  a  process  improvement  plan  and  to  support  periodic 
appraisals  to  determine  improvement  progress) 

What  are  the  advantages  of  value  engineering  and  award  fees?  Exercising  the  value  engi¬ 
neering  clause  could  have  a  greater  financial  reward  potential  than  an  award  fee.  In  addition: 

•  An  award  fee  requires  an  increase  in  obligation  authority;  value  engineering  does  not. 

•  Value  engineering  requires  visibility  into  systems/software  engineering  work  packages 
and  pricing;  award  fee  application  of  SCAMPI  does  not. 

Ultimately,  an  organization  exercising  the  value  engineering  clause  has  the  potential  to  dem¬ 
onstrate  that  the  systems/software  engineering  process  improvement  instantiated  the  resulting 
cost  savings  as  well  as  value  added  to  the  products  produced  for  the  sponsoring  organization. 

The  bottom  line  in  the  brief  discussions  of  award  fee  and  value  engineering  is  that  both  in¬ 
centive  approaches  help  management  (sponsoring  organization  and  contractor)  to  focus  on 
overall  process  improvement. 
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SCAMPI  Activities  that  Support  Source  Selection 

Using  SCAMPI  to  Establish  Baseline  Performance 

The  following  tables  provide  the  essentials  of  SCAMPI  V 1 . 1  and  corresponding  source  se¬ 
lection  activities.  Much  of  the  material  in  these  tables  duplicates  information  from  elsewhere 
in  this  document,  but  is  included  here  to  give  a  brief,  global  view  of  SCAMPI  relative  to 
typical  source  selection  activities. 


Table  111-5:  Analyze  Requirements  SS-Specific  Activities 


Table  111-6:  Develop  Appraisal  Plan  SS-Specific  Activities 
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Table  111-7:  Select  and  Prepare  Team  SS-Specific  Activities 


1.3  Select  and  Prepare  Team 


Purpose 


Ensure  that  an  experi¬ 
enced,  trained,  and  ap¬ 
propriately  qualified 
team  is  available  and 
prepared  to  execute  the 
appraisal  process. 


Inputs 


Appraisal  requirements 
and  constraints 

Appraisal  plan 

CMM  scope 

Team  training  materials 


Outputs 


Training  records 

Team  leader  selections 

Team  member  assign¬ 
ments  and  qualifications 

Prepared  team  that  has 
completed 

•  method  training 

•  model  training 

•  team-building  activi¬ 
ties 

•  team  orientation 
regarding  appraisal 


Source-Selection- 
Specific  Activities 


Select,  train,  and  prepare 
SCAMPI  personnel  with 
acquisition  requirements 
in  context. 


Table  ill-8:  Obtain  and  Analyze  Initial  Objective  Evidence  SS-Specific  Activities 


1.4  Obtain  and  Analyze  Initial  Objective  Evidence 


Purpose 


Obtain  information  that 
facilitates  site-specific 
preparation  and  an  un¬ 
derstanding  of  the  im¬ 
plementation  of  model 
practices  across  the  or¬ 
ganizational  unit.  Iden¬ 
tify  potential  issues, 
gaps,  or  risks  to  aid  in 
refining  the  plan. 
Strengthen  the  under¬ 
standing  of  the  organi¬ 
zation’s  operations  and 
processes. 


Inputs 


Practice  implementation 
data  for  the  organiza¬ 
tional  unit 

Identified  participants 

Participants  that  are 
briefed  and  oriented  on 
appraisal  activities 


Outputs 


Completed  instruments 

Data  analysis  results 
(data  summaries,  ques¬ 
tionnaire  results,  etc.) 

Identification  of  addi¬ 
tional  information  needed 

Prepared  participants 

Initial  set  of  objective 
evidence 


Source-Selection- 
Specific  Activities 


Receive  proposals. 

Evaluate  the  proposals 
initiated. 

Determine  responsive¬ 
ness  of  offerors  (e.g., 
Quick  Look). 

Analyze  offerors’ 
SCAMPI  information 
for  establishing  “gen¬ 
eral”  prioritization  of 
reference  model  compo¬ 
nents  for  all  offerors 
relative  to  objectives  of 
the  acquisition.* 

Finalize  logistical  coor¬ 
dination  for  site  visits.* 


*  These  SCAMPI  activities  will  have  timing  and  scheduling  components  tied  directly  to  the  overall  acquisition 
schedule. 
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Table  111-9:  Prepare  for  Collection  of  Objective  Evidence  SS-Specific  Activities 


1.5  Prepare  for  Collection  of  Objective  Evidence 


Purpose 


Plan  and  document  spe¬ 
cific  data  collection 
strategies,  including 
sources  of  data,  tools, 
and  technologies  to  be 
used,  and  contingencies 
to  manage  the  risk  of 
insufficient  data. 


Inputs 


Appraisal  plan 

Process  implementation 
indicators  (PIIs)  for  the 
organizational  unit 

Initial  objective  evidence 
review 

Data  collection  status 


Outputs 


Confirmation  that  objec¬ 
tive  evidence  collected  is 
sufficient  to  proceed 

Initial  data  collection 
plan 

Replan  of  data  collection 


Source-Selection- 
Specific  Activities 


Depending  on  the  timing 
of  SCAMPI  activities 
and  the  source  selection 
schedule,  Quick  Look 
determinations  could  be 
supported  with  initial 
SCAMPI  data  analysis 
(readiness  review)  de¬ 
termined  during 
SCAMPI  process  1.4. 


Table  111-10:  Examine  Objective  Evidence  SS-Specific  Activities 


2.1  Examine  Objective  Evidence 


Purpose 


Collect  information 
about  the  practices  im¬ 
plemented  in  the  organi¬ 
zation  and  relate  the 
resultant  data  to  the  ref¬ 
erence  model.  Perform 
the  activity  in  accor¬ 
dance  with  the  data  col¬ 
lection  plan.  Take  cor¬ 
rective  actions  and 
revise  the  data  collection 
plan  as  needed. 


Inputs 


Appraisal  data: 

•  initial  objective  evi¬ 
dence  documents 

•  documented  practice 
implementation  gaps 

•  feedback  from  pre¬ 
liminary  findings 

Data  collection  plan: 

•  appraisal  schedule 

•  interview  schedule 

•  document  list 

•  new  interview  ques¬ 
tions 


Outputs 


Updated  appraisal  data 

An  updated  data  collec¬ 
tion  plan 


Source-Selection- 
Specific  Activities 


Continue  to  evaluate  the 
proposals. 

Initiate  the  SCAMPI  on 
site  for  each  offeror. 
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Table  111-11:  Verify  and  Validate  Objective  Evidence  SS-Specific  Activities 


2.2  Verify  and  Validate  Objective  Evidence 

Purpose  Inputs  Outputs 


Source-Selection- 
Specific  Activities 


Updated  appraisal  data: 

•  notes 

•  strength/weakness 
statements 

•  annotated  worksheets 

Updated  appraisal  arti¬ 
facts: 

•  preliminary  findings 

•  revised  data  collection 
plan 

Requests  for  additional 


Verify  the  implementa-  Appraisal  plan:  Updated  appraisal  data:  Continue  to  evaluate 

tion  of  the  organiza-  #  schedule  and  partici-  •  notes  proposals. 

•  ^ength/weakness  Continue  the  SCAMPI 

date  the  preliminaty  _  .  .  StatementS  on  site  for  each  offeror 

findings  describing  gaps  ^  *  annotated  worksheets  Preliminary  findings 

in  the  implementation  of  Updated  appraisal  arti-  presentations  or  focus 

model  practices.  Each  *  strength  and  weakness  facts:  group  interviews  are 

implementation  of  each  statements  .  preliminaiy  findings  optional,  but  recom- 

practice  is  verified  so  Data  collection  plan:  .  ,  ,  ..  .  mended,  practices  m  a 

F  •  revised  data  collection  qCAMPT  nnnrakal  * 

that  it  may  be  compared  •  specifying  additional  p]an  PP 

to  CMMI  practices,  and  information  needed  „  , .  ,  Suggested  techniques 

the  team  characterizes  Requests  for  additional  are  optional  but  the 

the  extent  to  which  the  data  validation  of  prelimi- 

practices  in  the  model  nary  flndin8s  in  some 

are  implemented.  Gaps  manner  is  a  recluired 

in  practice  implementa-  SCAMPI  practice  to 

..  .  i  i  complete  the  appraisal, 

tion  are  captured  and  ^ 

validated  with  members 
of  the  organization.  Ex¬ 
emplary  implementa¬ 
tions  of  model  practices 
may  be  highlighted  as 
strengths  to  be  included 
in  appraisal  outputs. 


*  The  decision  regarding  how  these  practices  will  or  will  not  be  executed  is  made  during  process  1 .2,  De¬ 
velop  Appraisal  Plan. 


Continue  to  evaluate 
proposals. 

Continue  the  SCAMPI 
on  site  for  each  offeror. 

Preliminary  findings 
presentations  or  focus 
group  interviews  are 
optional,  but  recom¬ 
mended,  practices  in  a 
SCAMPI  appraisal.* 

Suggested  techniques 
are  optional  but  the 
validation  of  prelimi¬ 
nary  findings  in  some 
manner  is  a  required 
SCAMPI  practice  to 
complete  the  appraisal. 
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Table  111-12:  Document  Objective  Evidence  SS-Specific  Activities 


2.3  Document  Objective  Evidence 


Purpose 


Create  lasting  records  of 
the  information  gath¬ 
ered,  by  identifying  and 
then  consolidating  notes 
and  transforming  the 
data  into  records  that 
document  practice  im¬ 
plementation  as  well  as 
strengths  and  weak¬ 
nesses. 


Inputs 


Appraisal  data: 

•  notes  taken  during 
data  collection  activi¬ 
ties 

•  annotated  worksheets 
or  other  work  aids 
containing  data 

•  strengths  and  weak¬ 
nesses  documented 
from  previous  activi¬ 
ties 

•  data  collection  plan 


Outputs 


Updated  appraisal  data: 

•  noted  practice  imple¬ 
mentation  gaps 

•  revised  data  collection 
plan 

•  annotated  worksheets 

Requests  for  additional 
data  (interviewees  or 
documents) 


Source-Selection- 
Specific  Activities 


Continue  to  evaluate 
proposals. 

Continue  the  SCAMPI 
on  site  for  each  offeror. 


Table  111-13 :  Generate  Appraisal  Results  SS-Specific  Activities 


2.4  Generate  Appraisal  Results 

Purpose  Inputs 

Outputs 

Source-Selection- 

Specific  Activities 

Rate  goal  satisfaction  Appraisal  data: 

Final  findings 

Continue  to  evaluate 

based  upon  the  extent  of  .  validated  preliminary 

Recorded  rating  decisions 

proposals. 

practice  implementation  findings 

Continue  the  SCAMPI 

throughout  the  organi-  ,  ,  .  „  ,  . 

.  ,  .  ®  tabulations  of  objec- 

zational  unit.  The  extent  .  . ,  / 

r  ,  '  tive  evidence  of  prac- 

of  practice  implementa-  .  . 

.  .  ,  .  tice  implementation 

tion  is  deter-  r 

on  site  for  each  offeror. 

mined/judged  based  on  *  annotated  worksheets, 
validated  data  (e.g.,  di-  checklists,  and  work- 

rect,  indirect,  and  af-  notes 

firmation  objective  evi¬ 
dence)  collected  from 
the  entire  representative 
sample  of  the  organiza¬ 
tional  unit.  The  rating  of 
capability  levels  and/or 
maturity  levels  is  driven 
by  the  goal  satisfaction 
ratings. 
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Table  111-14:  Deliver  Appraisal  Results  SS-Specific  Activities 


3.1  Deliver  Appraisal  Results 

Purpose  Inputs 


Provide  credible  ap¬ 
praisal  results  that  can 
be  used  to  guide  actions. 

Represent  the  strengths  '  Recommendations  repott  on  site  for  each  offeror, 

and  weaknesses  of  the  Appraisal  artifacts.  requested^  (The  source  selection 

processes  in  use  at  the  •  appraisal  input  process  and  constraints 

time.  Provide  ratings  (if  •  appraisal  plan  may  prevent  the  delivery 

planned  for)  that  accu-  of  appraisal  results  on 

rately  reflect  the  capa-  site.  If  that  is  the  case, 

bility  level/maturity  provide  the  appraisal 

level  of  the  processes  in  results  at  a  later  time  as 

use.  required  by  the 

SCAMPI  method.) 

Incorporate  the  appraisal 
results  into  the  source 
selection  evaluation 
context  to  be  presented 
to  the  SSAC  and  SSA. 

The  SSEB  compares 
data  collected  against 
the  evaluation  standard 
and  assigns  technical 
ratings  and  risk  identifi¬ 
cations. 

The  SSAC  compares 
and  ranks  offeror  pro¬ 
posals  and  submits  a 
risk  assessment  to  the 
SSA. 

The  SSA  makes  the 
award  decision. 


Appraisal  data: 

•  final  findings 

•  ratings 
Appraisal  artifacts: 

•  appraisal  input 

•  appraisal  plan 


Documented  final  find- 


Source-Selection- 
Specific  Activities 


Continue  to  evaluate 
proposals. 


Final  report  (if  requested)  Continue  the  SCAMPI 


Recommendations  report 
(if  requested) 


The  SCAMPI  appraisal  team  may  or  may  not  be  formally  part  of  the  SSEB.  If  they  are  not,  the  team  provides 
SCAMPI  findings/outcomes  to  the  SSEB.  The  appraisal  team  consults  with  the  SSEB  if  requested.  The  appraisal  team 
may  act  as  advisors  to  the  SSAC  and  SSA. 
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Table  III- 1 5:  Package  and  Archive  Appraisal  Results  SS-Specific  Activities 


3.2  Package  and  Archive  Appraisal  Results 


Purpose 


Preserve  important  data 
and  records  from  the 
appraisal,  and  dispose  of 
sensitive  materials  in  an 
appropriate  manner. 


Inputs 


Appraisal  data: 

•  appraisal  input 

•  appraisal  plan 

•  final  findings 

•  objective  evidence 
Appraisal  team  artifacts: 

•  notes 

•  documented  practice 
implementation  gaps 

•  preliminary  findings 

•  document  library 


Outputs 


Appraisal  record 

Completed  forms  and 
checklists 

Sanitized  data  (as  appro¬ 
priate  and  agreed  upon 
during  planning) 

Lessons  learned  (ap¬ 
praisal  team,  organiza¬ 
tion) 


Source-Selection- 
Specific  Activities 


The  appraisal  record 
will  be  tailored  and 
sanitized  as  appropriate 
and  as  agreed  to  during 
planning. 


Reduced  Risk  Objectives 

The  primary  reason  for  using  SCAMPI  in  source  selection  is  to  reduce  the  risk  of  selecting  an 
organization  that  has  immature  and  ineffective  process  and  product  life  cycles.  This  risk  re¬ 
duction  activity  enhances  the  achievement  of  successful  system/product  development  and 
delivery  to  the  customer.  Using  SCAMPI  in  contract  process  monitoring  enables  the  cus¬ 
tomer  to  have  detailed  insight  and  incentive  tools  available  to  ensure  that  a  development  or¬ 
ganization  maintains  its  mature  processes  or  is  making  steady,  measured  progress  in  achiev¬ 
ing  defined  improvement  objectives  relative  to  the  CMMI  model.  While  using  SCAMPI  to 
benchmark  an  organization’s  process  and  product  life  cycles  does  not  necessarily  guarantee  a 
successful  product,  the  likelihood  of  success  should  increase  as  the  processes  mature.  In  other 
words,  mature  processes  reduce  the  risk  associated  with  the  planned  product  development. 
Reduced  risk  is  the  benefit. 


Schedule  and  Resource  Issues 

One  of  the  major  issues  related  to  implementing  SCAMPI  in  source  selection  is  the  compati¬ 
bility  of  the  source  selection  schedule  with  the  SCAMPI  appraisal  schedule.  The  typical 
SCAMPI  evaluation  takes  approximately  10  working  days  for  the  site  visit  alone  (offeror 
preparation  work  and  potential  pre-on-site  visit  preparation  requirements  exacerbate  the 
schedule). 


An  acquisition  reform  push  over  the  last  several  years  has  significantly  reduced  source  selec¬ 
tion  schedules  (90  days,  120  days,  etc.).  Given  the  lead  times  for  source  selection  briefing 
generation  and  coordination  prior  to  presenting  to  the  SSA,  even  less  time  is  available  for  the 


CMU/SEI-2002-HB-002 


111-47 


actual  evaluation.  For  example,  consider  the  situation  where  a  planned  acquisition  has  two 
1-  idders  and  no  subcontractors.  Using  SCAMPI  for  source  selection  is  probably  feasible  in 
nominal  schedule  timeframes.  However,  some  larger  programs  can  have  five  offerors,  each 
having  three  major  subcontractors.  That  equates  to  thirty  weeks  of  site  visit  time.  The  use  of 
SCAMPI  and  scheduling  site  visits  becomes  complex  to  meet  typical  source  selection 
timeframe  requirements. 

A  balance  must  be  established  between  the  source  selection  schedule  and  the  coverage  to  be 
provided  in  the  SCAMPI  evaluation.  For  example,  the  following  are  ways  that  could  be  used 
to  accommodate  the  SCAMPI  evaluation  in  an  otherwise  short  source  selection  schedule: 

•  The  SCAMPI  evaluation  could  be  tailored  to  look  at  only  a  subset  of  the  process  areas. 

•  Multiple  SCAMPI  teams  could  operate  in  parallel. 

•  Evaluating  subcontractors  could  be  excluded  from  the  SCAMPI  evaluation  (with  the  at¬ 
tendant  increase  in  risk). 

•  The  source  selection  schedule  could  be  established  to  accommodate  the  level  of  SCAMPI 
thoroughness  required. 

•  The  SCAMPI  team  could  be  excluded  from  the  SSEB  membership.  That  allows  evalua¬ 
tions  to  start  prior  to  opening  discussions.  This  accommodation,  of  course,  decreases  the 
level  of  insight  and  participation/input  that  the  SCAMPI  team  has  in  the  rest  of  the 
evaluation. 

•  The  full  process  area  coverage  desired  could  be  kept,  but  an  ARC  Class  B  or  Class  C  ap¬ 
praisal  could  be  performed. 

•  Reuse  the  results  of  prior  process  appraisals  of  the  supplier  in  lieu  of  performing  a  sepa¬ 
rate  SCAMPI  appraisal.  (See  the  discussion  on  appraisal  reuse  on  page  61.) 


There  are  probably  other  innovative  ways  to  address  schedule  imbalance,  and  each  program 
has  its  own  particular  issues  to  address  and  accommodate.  It  is  important,  though,  to  recog¬ 
nize  early  that  there  may  be  a  schedule  issue  that  affects  your  program  and  that  you  may  need 
to  devise  a  SCAMPI  approach  that  balances  the  benefits  of  a  shortened  source  selection 
timeline  and  SCAMPI  risk  reduction. 

Criteria  for  Using  SCAMPI  in  the  Acquisition  Environment 

Determining  How  to  Use  SCAMPI 

Most  organizations  selecting  suppliers  of  systems  and  software  products  and  services  would 
prefer  to  have  more  discriminators  available  other  than  cost  and/or  the  technical  performance 
promised  in  a  proposal.  Use  of  SCAMPI  can  provide  such  a  discriminator  among  the  various 
systems  and  software  development  organizations  that  are  under  consideration.  Table  III-6  on 
page  41  and  Figure  III-2  on  page  50  demonstrate  the  need  for  some  analysis  of  the  require¬ 
ments  at  the  outset.  Note  that  the  decision  diamond  in  Figure  III-2  labeled  “Source  Selection, 
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Teaming,  Subcontract”  considers  three  acquisition  applications.  The  sections  that  follow  will 
discuss  SCAMPI  with  respect  to  a  “yes”  answer  to  this  decision  diamond  in  relation  to  the 
perspective  of  deciding  to  use  or  not  use  the  SCAMPI  Method.  The  “no”  option  leading  to  a 
“yes”  regarding  incentivization  will  be  discussed  separately. 


Source  Selection:.  This  typically  has  referred  to  the  application  in  which  a  U.S.  Government 
agency  (e.g.,  Army,  Navy,  Marine  Corps,  Air  Force,  Coast  Guard,  Federal  Aviation  Admini¬ 
stration)  uses  SCAMPI  as  a  contract  award  evaluation  criterion.  For  a  company  contemplat¬ 
ing  the  selection  of  suppliers  of  systems  engineering/software  services  or  products  (non- 
Govemment  procurement),  many  of  the  goals,  objectives,  constraints,  strategies,  and  plan¬ 
ning  factors  are  germane  and  should  be  considered. 


Teaming:  The  teaming  situation  refers  to  the  industiy  practice  of  selecting  partners  or 
“teammates”  to  collaboratively  compete  for  contracts,  Government  or  commercial.  This  is  a 
variation  of  the  source  selection  application  use  of  SCAMPI.  For  this  application,  the  degree 
of  legal  formality  may  typically  be  reduced.  However,  the  considerations  for  selecting  such  a 
“teammate”  remain  essentially  the  same. 

Subcontract:  The  subcontracting  instance  refers  to  the  situation  in  which  one  entity  (e.g., 
Government  prime  contractor  or  commercial  entity)  establishes  a  relationship  with  another 
entity  for  the  purpose  of  receiving  services  or  products  that  will  then  be  applied  to  an  existing 
effort.  This  could  take  the  form  of  the  primary  organization  “subcontracting”  a  specified  por¬ 
tion  of  an  existing  contract  or  statement  of  work,  or  separately  identified  services  that  will 
supplement  the  primary  organization’s  own  efforts  on  a  contract  or  product. 
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Figure  111-2:  SCAMPI  Acquisition  Implementation  Options 


111-50 


CMU/SEI-2002-HB-002 


Criteria  for  Using  SCAMPI  in  Source  Selection 


General  familiarity  with  the  acquisition  organization  source  selection  process  is  assumed  for 
the  purpose  of  focusing  on  each  participant’s  (evaluator’s  or  recipient’s)  relationship  to 
SCAMPI.  Clearly,  SCAMPI  should  not  be  used  for  every  acquisition.  Both  the  costs  and 
benefits  of  using  SCAMPI,  as  well  as  the  specific  nature  of  the  acquisition,  should  be  consid¬ 
ered  when  making  this  decision.  These  costs  and  benefits  may  indicate  that  other  approaches 
are  necessary  for  very  small  acquisitions  involving  systems  and  software.  This  section  dis¬ 
cusses  criteria  related  to  the  nature  of  an  acquisition. 

Several  considerations  must  be  weighed  by  a  program  manager  when  making  the  decision. 
All  of  the  following  factors  are  important  considerations,  but  the  program  manager  or  person 
responsible  for  determining  SCAMPI  usage  for  an  acquisition  must  weigh  them  in  accor¬ 
dance  with  the  organization’s  method  of  procuring  systems  and  services.  These  are  general 
guidelines  that  must  be  refined  to  meet  the  context  of  the  organization: 

•  criticality  of  an  acquisition 

•  available  data  about  the  offeror’s  past  performance  or  product  capability 

•  total  dollar  value  of  the  acquisition 

•  management  control  priority 

•  unprecedented  system  mission  needs 

•  acquisition  life-cycle  phase 

•  length  of  acquisition  time  period 

•  prime  contractor-subcontractor  relationship 

Table  III- 16  illustrates  the  relationship  of  each  of  these  factors  as  a  general  guideline  for  de¬ 
termining  the  appropriateness  of  SCAMPI  usage.  Each  box  should  be  read  independently  and 
then  combined  with  other  factors  to  make  an  overall  judgment  on  SCAMPI  applicability. 
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Table  111-16:  SCAMPI  Usage  Decision-Making  Criteria 


Decision 

Definitely 

Strongly 

Consider 

SCAMPI  Use 

Use  SCAMPI 

Consider 

Using  SCAMPI 

Likely  Not 

Using  SCAMPI 

Appropriate 

Critical 

Systems/ 

Software 

DoD  ACAT  1 

Non-MCCR 

program/systems 
or  Government- 
determined 

systems 

“Major”  Program 

Management 

Control 

High  Priority 

Low  Priority 

Criteria 

Systems/ 

Software 

Precedence 

Unprecedented 

system 

Need  defined, 
any  Systems/ 
Software  CIs 
unprecedented 

Precedented 

system 

Life  Cycle 

Phase 

SDD  phase 

•  SDD 

•  CTD 

Operational 
readiness  support 

Production/ 

deployment 

Schedule 

Length 

Upgrades,  major 
modifications  or 
follow-ons 
expected 

•  Development 
>24  months 

•  System  life 
>10  years 

Program  length 
>  5  years 

Criticality  of  an  Acquisition  or  the  Systems/Software  Component 

The  criticality  of  an  acquisition  may  necessitate  SCAMPI  use.  It  is  recommended  that  any 
Government-defined  “major”  program  use  SCAMPI  as  an  integral  part  of  its  strategy  for 
producing  the  highest  quality  end  product  and  motivating  Government  contractors  to  focus 
on  systems/software  process  improvement  as  a  means  to  effect  this  goal.  In  all  Government- 
deemed  major  programs,  as  well  as  Mission  Critical  Computer  Resource  (MCCR)  systems, 
regardless  of  total  dollar  amount,  systems/software  size,  or  DoD  priority  ranking,  SCAMPI 
use  should  be  strongly  considered.  MCCR,  and  systems/software  in  general,  are  critical  com¬ 
ponents  of  modem  weapon  systems.  The  success  of  the  system  is  largely  dependent  upon  the 
systems/software  precisely  performing  its  intended  function.  An  example  of  a  small  but 
highly  critical  component  of  a  system  and  software  warranting  the  use  of  SCAMPI  in  an  ac¬ 
quisition  would  be  software  needed  to  control  the  hardware  for  an  access  control  system  for 
nuclear  weapons  or  other  munitions. 
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Management  Control  Priority 

When  management  control  is  a  high-priority  concern,  SCAMPI  use  should  be  considered.  An 
environment  under  effective  management  control  will  be  more  able  to  produce  data  that  is 
useful  for  lessons  learned  that  can  be  incorporated  into  the  overall  system  development  proc¬ 
ess.  These  lessons  help  the  acquisition  organization  avoid  reinventing  the  wheel.  Successful 
management  control  also  facilitates  effective  implementation  of  modem  methodologies, 
tools,  and  techniques. 

A  controlled  environment  is  essential  to  managing  contractor  processes — processes  for 
maintaining  the  development  environment,  bringing  new  people  and  technology  into  the  en¬ 
vironment,  identifying  problems  early  in  the  contract,  and  managing  requirements  changes.  A 
controlled  environment  enables  improved  risk  assessment  and  abatement. 

System/Software  Precedence 

SCAMPI  should  be  used  when  the  contractor  is  likely  to  develop  systems/software  imple¬ 
mentations  that  are  unprecedented.  Unprecedented  systems  (i.e.,  those  solving  new  or  unique 
problems)  pose  special  problems  for  systems/software  development  organizations.  A 
SCAMPI  of  each  offeror  would  identify  whether  the  requisite  controls  are  in  place  on  the 
contractors’  existing  programs  and  whether  they  will  be  easily  transferred  to  the  new,  un¬ 
precedented  system. 

When  the  mission  of  the  system/software  component,  especially  the  role  played  by  the  soft¬ 
ware  component,  is  known  and  defined  by  the  end  user,  and  portions  of  the  system  will  be 
unprecedented,  use  of  SCAMPI  should  be  strongly  considered.  SCAMPI  helps  identify  pro¬ 
gram  risks  associated  with  the  capability  of  contractors  to  succeed  in  producing  quality  sys¬ 
tems/software  in  an  unprecedented  environment. 

Use  of  SCAMPI  yields  information  about  an  organization’s  ability  to  manage  risks  inherent 
in  unprecedented  systems/software  development,  as  well  as  its  ability  to  manage  tasks  that 
are  new  but  are  similar  to  ones  it  has  successfully  completed  previously. 


Life-Cycle  Phase 

The  life-cycle  phase  of  an  acquisition  is  an  important  factor  in  determining  SCAMPI  usage. 
The  CMM  and  subsequent  appraisal  methods  were  originally  developed  in  response  to  the 
DoD’s  and  industry’s  recognized  problems  in  managing  the  development  of  increasingly 
complex  mission  critical,  systems/software-intensive  products  in  the  real-time,  embedded 
domain.  Given  this  background,  SCAMPI  fits  in  any  System  Development  &  Demonstration 
(SDD)  program  within  this  domain,  since  SDD  is  the  typical  phase  associated  with  major 
new  systems/software  development.  It  is  recommended  that  any  SDD  program  consider 
SCAMPI  use,  in  accordance  with  the  other  factors  listed  here.  However,  SCAMPI  use  is  not 
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limited  to  the  SDD  phase.  The  SCAMPI  method  could  be  used  successfully  in  the  Concept  & 
Technology  Development,  Production  &  Deployment,  and  Operations  &  Support  phases. 

Schedule  Length 

The  SCAMPI  Method  should  be  considered  in  any  procurement  in  which  the  system  de¬ 
sign/engineering  or  software  development  is  a  major  component  and  the  program  duration 
period  is  expected  to  be  greater  than  24  months.  This  time  frame  is  recommended  because  of 
the  resources  necessary  to  apply  SCAMPI  effectively  and  because  the  typical  process  im¬ 
provement  program  implemented  by  a  contractor  requires  at  least  18  to  24  months  to  attain 
and  sustain  improvements  in  process  maturity.  Thus,  more  systems/software  development 
time  is  necessary  to  see  improved  results  directly  on  the  contract. 


SCAMPI  should  also  be  used  when  the  program  office  expects  significant  block  upgrades, 
modifications,  or  follow-on  programs  to  occur,  and  the  original  contractor  is  expected  to  be  a 
primary  offeror  or  likely  performer  of  the  new  work.  Often  the  processes  put  in  place  by  the 
contractor  at  the  start  of  a  systems/software  development  will  be  frozen,  meaning  that  process 
changes  will  be  limited  during  that  development  period.  Systems/software  upgrades  or  major 
modifications  to  existing  systems  are  good  times  to  unfreeze  the  current  sy  stems/ soft  ware 
development  process  and  install  new,  improved  processes,  methods,  and  technology.  There¬ 
fore,  using  SCAMPI  during  the  initial  systems/software  development  and  the  subsequent  im¬ 
provement  programs  will  enable  any  improved  processes  to  be  implemented  on  the  follow-on 
developments. 

SCAMPI  use  may  still  be  appropriate  even  if  neither  of  these  criteria  is  met  and  the  Govern¬ 
ment  Program  Executive  Officer  (PEO),  center  commander,  or  activity  committee  is  at¬ 
tempting  to  motivate  and  gain  improvements  in  a  particular  domain  area,  such  as  avionics 
systems.  These  PEO  decisions  may  entail  long-range  considerations  that  go  beyond  the  cur¬ 
rent  contract,  and  thus  SCAMPI  use  may  be  appropriate  to  meet  other  Government  objec¬ 
tives. 

Benefits  of  Using  SCAMPI  in  Source  Selection 

Use  of  appraisals  and,  by  extension,  the  SCAMPI  Method  in  Army,  Navy,  Air  Force,  and 
non-DoD  agencies  indicates  that  SCAMPI  can  help  the  acquiring  organization  in  multiple 
ways: 

•  added  systems/software  development  capability  realism  in  the  source  selection  process 

•  increased  objectivity  in  information  collected  for  an  acquisition 

•  motivation  for  contractor  systems/software  process  improvement  actions 


Systems/Software  Development  Capability  Realism:  One  benefit  SCAMPI  provides  is  the 
systems/software  development  capability  realism  introduced  into  the  proposal  review  and 
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contractor  analysis  process.  The  information  SCAMPI  collects  is  timely  and  real,  and  is 
based  on  current  projects  and  the  practices  actually  being  implemented  by  offerors’  engi¬ 
neering  and  managerial  personnel. 

For  moderate  to  large  systems/software  development  efforts,  a  currently  popular  means  of 
evaluating  a  contractor’s  systems/software  development  abilities  during  a  source  selection  is 
the  review  of  the  offeror’s  Systems  Engineering  Management  Plan  (SEMP)  and  Software 
Development  Plan  (SDP).  Comparing  the  SCAMPI  findings  with  the  evaluation  of  the  con¬ 
tractor’s  proposal  and  SEMP/SDP  will  clarify  for  the  program  office  whether  the  proposed 
approach  is  realistic  in  light  of  the  offeror’s  current  process  capability.  Based  on  this  com¬ 
parison,  the  program  office  can  better  evaluate  the  risks  posed  by  each  offeror  and  work  with 
the  successful  offeror  on  a  realistic  systems/software  process  improvement  program. 


Objectivity:  A  second  major  benefit  of  SCAMPI  is  the  objectivity  it  introduces  into  the  pro¬ 
posal  review  process.  The  SCAMPI  Method  helps  ensure  an  objective  review  by  putting  a 
trained  appraisal  team  on  site  to  evaluate  the  offeror’s  activities  and  compare  them  against  a 
public  reference  model  or  standard  (e.g.,  the  CMMI).  In  the  typical  source  selection,  evalu¬ 
ating  systems/software  risk  is  a  difficult  task  because  there  are  few  avenues  for  addressing 
this  issue  other  than  by  evaluating  what  is  in  the  proposal. 

With  the  goals  of  the  CMMI  PAs  as  a  basis,  contractor  systems/software  process  capability 
can  be  reliably  measured  against  a  common  standard.  This  permits  consistent,  repeatable,  and 
fair  evaluation  of  contractor  systems/software  process  capability.  This  adds  value  to  the 
source  selection  process  by  making  systems/software  reviews  more  objective. 

Cost  of  Using  SCAMPI 

Using  SCAMPI  requires  personnel  and  financial  resources,  on  both  the  contractor  and  acqui¬ 
sition  agency  sides.  The  resource  considerations  affecting  the  implementation  of  SCAMPI  are 

•  personnel 

•  time 

•  scope  of  appraisal  (scope  of  the  organizational  unit  and  scope  of  the  model) 

•  financial 

•  development  organization’s  resource  requirement 

Figure  III-3  illustrates  the  estimated  acquisition  agency  labor,  in  person  days,  required  to 

•  implement  SCAMPI  in  program  documentation 

•  train  SCAMPI  team  members  and  conduct  a  pre-on-site  of  contractor  preparedness 

•  conduct  the  site  visits 

•  incorporate  the  SCAMPI  findings  into  source  selection  results/decisions 
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The  estimate  assumes  a  shgle  source  selection,  a  program  office  having  no  prior  experience 
with  SCAMPI,  and  three  offerors  deemed  responsive  who  must  be  evaluated  serially.  For  a 
single  team  of  five  who  conduct  three  site  visits,  the  total  labor  is  210  person  days.  For  refer¬ 
ence,  the  estimated  labor  for  an  acquisition  involving  only  one  site  visit  is  98  person  days 
(Total  Effort  Fixed  Costs  42  person  days  plus  Variable  Cost  Effort  of  56  person  days  for  site 
visits).  Certainly,  there  are  economies  of  scale  and  there  are  many  non-recurring  costs,  such 
as  team  training,  which  will  continue  to  reduce  overall  acquisition  agency  labor  costs  as 
trained  resources  are  utilized  on  subsequent  acquisitions. 

The  nominal  on-site  time  for  each  contractor  would  be  approximately  three  weeks  of  interac¬ 
tion  spread  over  a  two-month  calendar  period  of  time.  Clearly,  early  planning  and  decisions 
upon  execution  of  the  SCAMPI  are  required  due  to  the  sheer  length  of  time  involved  with 
just  a  single  contractor. 

This  analysis  leaves  it  up  to  members  of  the  individual  program  office  to  determine  their  own 
average  person  cost  per  day  and  average  travel  and  per  diem  costs,  and  subsequently  add 
these  to  the  cost  of  team  training  to  estimate  a  total  dollar  cost  for  implementing  SCAMPI. 


I  SCAMPI  Effort 

I  Phase 

Who 

Does  It 

Effort  days 
per  person 

Number  of 
People 

Total  ] 

Effort 

SCAMPI  Information 
gathering 

SCAMPI  Lead  Ap¬ 
praiser,  PM,  CO 

2 

3 

6 

RFP  Preparation 

SCAMPI  Lead  Ap¬ 
praiser  or  PM 

3 

1 

3 

SCAMPI  Training 

SCAMPI  team 
members 

5 

5 

25 

Fixed 

Costs 

SCAMPI  Findings 
Preparation 

SCAMPI  team 
members 

1 

5 

5 

Contractor  Debriefs 

PM,  CO,  SCAMPI 
team  leader 

1 

3 

3 

Subtotals 

12 

42 

Variable 

Cost 

Pre-on-site 

Visits  (3) 

Selected  SCAMPI 
team  members  (2) 

3 

2 

18 

3  SCAMPI  Site 

Visits  (offerors) 

SCAMPI  team 
members 

10 

5 

150 

Total  Person  Days  Effort 

210 

Figure  111-3:  Estimated  SCAMPI  Labor  for  One  Source  Selection 


Constraints 

Personnel  Constraints:  The  largest  constraint  on  the  acquisition  agency  is  the  labor  effort 
expended  by  the  individuals  constituting  the  SCAMPI  team.  Preparation  involves  the  acqui¬ 
sition  documents  preparation  and  several  days  per  offeror  to  ensure  that  they  are  prepared  to 


111-56 


CMU/SEI-2002-HB-002 


undergo  the  SCAMPI.  That  being  established,  this  team  is  needed  for  a  full  two  work  weeks 
for  every  SCAMPI  site  visit  that  is  performed.  In  addition,  several  person  days  are  needed  to 
prepare  the  detailed  report  or  set  of  findings  that  is  submitted  to  the  management  body  spon¬ 
soring  the  evaluation. 

In  addition  to  the  site  visit  requirements,  the  SCAMPI  team  leader  or  the  program  office’s 
systems/sofiware  point  of  contact  (POC)  will  be  needed  on  a  part-time  basis  prior  to  the  site 
visits  to  incorporate  appropriate  language  into  the  source  selection  materials  that  are  affected 
by  SCAMPI,  assemble  a  SCAMPI  team,  and  schedule  training  for  any  untrained  team  mem¬ 
bers.  This  part-time  task  will  be  minimal  once  the  acquisition  organization  has  put  in  place 
support  materials  for  SCAMPI,  including  this  guide.  After  the  site  visits,  the  SCAMPI  team 
leader  will  likely  be  needed  to  advise  the  evaluation  sponsor  and  perform  outbriefs  to  the  de¬ 
velopment  organizations  as  directed  by  his  or  her  Contracting  Officer. 

Time  Constraints:  The  trained  SCAMPI  team  is  needed  for  at  least  two  and  a  half  weeks  for 
every  site  visit.  This  includes 

•  site  readiness  preparation:  2-3  days 

•  travel  time:  1  day 

•  site  visit:  5-10  days  (includes  caucus  and  findings  preparation) 

•  time  off  between  site  visits:  1  week 

Time  off  is  important  because  site  visits  are  intense  activities.  Another  time  constraint  is  im¬ 
posed  by  the  typical  source  selection  schedule.  Site  visits  nominally  do  not  begin  until  after 
the  initial  proposal  evaluation  and  only  to  those  offerors  determined  to  be  responsive.  This 
typically  allows  a  one-to-two-month  window  to  conduct  the  on-site  phases  of  the  SCAMPI.  A 
program  manager  does  not  know  the  number  of  offerors  until  proposals  are  received.  This 
means  that  the  program  manager  will  have  to  estimate  how  much  time  is  needed  to  complete 
all  the  SCAMPI  appraisals  based  on  the  estimated  number  of  offerors. 

Suggestions  For  Optimizing  Resources 

Acquisition  organizations  performing  SCAMPI  appraisals  routinely  for  multiple  acquisitions 
can  optimize  the  resources  required  for  SCAMPI  implementation  by  assigning  full-time 
SCAMPI  support.  This  option  offers  the  greatest  savings  in  both  cost  and  personnel.  Full¬ 
time  support  could  take  the  form  of  dedicated  personnel  within  the  organization  or  from  a 
contracted  organization.  Dedicated  support  can  take  on  two  levels  of  involvement.  Personnel 
can 

•  help  with  the  source  selection  documentation  needed  to  use  SCAMPI,  identify  team 
members,  and  coordinate  their  training 

•  augment  the  SCAMPI  teams  for  specific  acquisitions  by  participating  in  the  on-site  visits 
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Fully  dedicated  personnel  who  have  already  come  down  a  SCAMPI  learning  curve  should  be 
capable  of  implementing  local  SCAMPI  policies  and  procedures  quickly  and  effectively, 
which  should  reduce  overall  costs. 

The  use  of  full-time  resources  to  augment  a  program’s  SCAMPI  team  will  ensure  organiza¬ 
tional  consistency  in  the  practice  of  the  SCAMPI  Method  and  assist  the  training  of  personnel 
through  a  form  of  on-the-job  technology  transition.  Utilizing  at  least  one  full-time  resource 
will  act  as  a  significant  acquisition  “force  multiplier”  when  it  comes  to  implementing 
SCAMPI  in  an  organization. 

The  following  approaches  to  cost  reduction  should  be  avoided  under  all  circumstances  be¬ 
cause  they  would  not  follow  the  SCAMPI  Method: 

•  not  training  team  members 

•  using  questionnaire  responses  and  or  Practice  Identification  Indicator  matrices  alone 
without  performing  site  visits 

These  approaches  undermine  the  consistency,  repeatability,  and  reliability  of  the  SCAMPI 
Method. 

Recipient 

From  the  recipient’s  view,  the  decision  to  use  SCAMPI  is  a  given.  The  recipient  organization 
will  be  aware  of  the  acquisition  organization’s  ultimate  decision  to  use  SCAMPI.  As  a  result, 
the  recipient  must  understand  what  the  decision  to  use  SCAMPI  actually  means  to  the  orga¬ 
nization.  The  U.S.  Government  has  been  steadfast  in  its  support  of  total  quality  management 
principles  and  the  belief  that  concentrating  on  systems/software  process  improvement  will 
pay  dividends  in  the  form  of  better,  less  costly,  shortened-time-to-customer/market  sys¬ 
tems/software  systems. 

To  remain  competitive  on  successive  acquisitions,  development  organizations  must  improve 
their  systems/software  development  processes.  In  contract  monitoring,  SCAMPI  can  augment 
a  process  improvement  action  plan  by  measuring  progress  relative  to  the  process  capabilities 
measured  during  source  selection.  The  Government’s  Performance  Risk  Analysis  Group 
(PRAG)  can  evaluate  process  improvement  based  on  past  performance  risk  assessments  of 
the  systems/software  process. 

By  making  SCAMPI  a  discriminator  in  conducting  acquisitions,  program  offices  will  moti¬ 
vate  contractors  to  focus  on  systems/software  process  capability  as  a  means  of  retaining  or 
increasing  acquisition  agency  business.  Given  the  premise  that  product  quality  will  follow 
process  quality,  focusing  on  systems/software  process  improvements  resulting  in  increased 
process  maturity  will  increase  the  likelihood  of 

•  accurate  estimates 
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•  decreased  variance  among  projects 

•  reduced  cost  and  schedule  targets 

Although  there  is  no  definitive  study  validating  these  benefits  quantitatively,  there  is  signifi¬ 
cant  anecdotal  evidence  from  individual  Government  and  industry  organizations  to  suggest 
that  these  benefits  are  real. 

A  focus  on  improving  systems/software  process  capability  should  lead  to  error  prevention, 
earlier  detection  of  errors  in  the  life  cycle,  and  an  ability  to  manage  a  requirements  change 
process,  producing  a  higher  quality  product  and  saving  the  acquisition  agency  money  over 
the  life  cycle  of  major  systems. 

Development  Organization  Resource  Requirements 

The  costs  of  SCAMPI  to  the  development  organization  are  significant — although  the  imme¬ 
diate  costs  are  not  always  as  high  as  those  of  the  acquisition  agency/organization  performing 
multiple  SCAMPIs  on  multiple  offerors.  A  company  will  expend  considerable  preparation 
time  for  SCAMPI.  The  company  is  typically  trying  to  put  its  best  foot  forward  for  the  acqui¬ 
sition  agency,  especially  if  the  SCAMPI  is  done  in  conjunction  with  a  source  selection.  Thus, 
all  development  organizations  will  perform  some  preparatory  tasks  to  accommodate  a 
SCAMPI.  Note  that  this  preparatory  time  is  exclusive  of  the  company’s  overall  process  im¬ 
provement  program,  which  may  span  a  number  of  years.  With  existing  Government  policy 
requiring  evaluations  against  specific  capability  maturity  levels  and  subsequent  risk  mitiga¬ 
tion,  most  organizations  will  have  embarked  on  a  process  improvement  program  of  their  own 
volition  to  remain  competitive  in  the  Government  sector  for  continued  work  as  well  as  re¬ 
maining  competitive  in  their  respective  commercial  sectors. 

Table  III- 17  provides  an  estimate  of  the  contractor  cost  for  supporting  a  SCAMPI  evaluation. 
The  table  does  not  capture  all  of  a  contractor  organization’s  efforts  from  the  beginning  of  a 
process  improvement  initiative  involving  CMMI.  The  table  is  presented  from  the  perspective 
of  an  organization’s  management  that  has  realized  that  they  desire  a  SCAMPI  evaluation  or 
their  organization  will  be  subject  to  SCAMPI  evaluation  and  the  approximate  person  effort 
and  calendar  time  anticipated  for  completing  an  ARC  Class  A  SCAMPI  evaluation.  In  con¬ 
structing  the  table,  one  assumption  was  that  the  appraisal  would  proceed  despite  the  results  of 
the  SCAMPI  pre-onsite  readiness  review,  which  determines  whether  the  SCAMPI  evaluation 
will  proceed  using  a  verification-based  approach  or  a  discovery-based  approach  (see  activity 
1.4.3,  Obtain  Initial  Evidence,  and  “Verification  vs.  Discovery”  in  the  “SCAMPI  Method 
Overview”  section  of  Part  I).  The  SCAMPI  method  allows  for  the  possibility  that  a  determi¬ 
nation  to  delay  the  SCAMPI  evaluation  may  be  made  based  on  the  results  of  the  readiness 
review.  This  would  most  likely  not  be  feasible  in  most  Government  source  selection  sched¬ 
ules  and  is  not  accounted  for  in  the  table. 
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The  preparation  time  of  four  to  six  person  effort  weeks  accounts  for  one  person  for  four  to  six 
weeks  or  larger  numbers  of  people  operating  in  parallel  for  shorter  durations  prior  to  the 
SCAMPI  activities  initiation.  The  table  uses  ges  due  to  the  variable  sizes  of  contractor 
organizations  and  the  resulting  range  of  time  projected  to  prepare.  Activities  include  identi¬ 
fying  programs  and  projects  for  review  and  verifying  their  current  maturity/capability  levels 
and  suitability  for  a  SCAMPI.  Once  projects  have  been  selected,  considerable  effort  would  be 
expended  ensuring  that  the  projects  have  submitted  documents  and  selected  interviewees  are 
prepared  for  the  SCAMPI.  This  could  include  “dry-run”  interviews,  organizing  documents, 
and  matrices  mapping  the  company’s  processes  to  the  reference  model.  For  an  organization 
undergoing  a  SCAMPI  for  a  Government  source  selection  decision  it  would  not  be  unheard 
of  for  the  organization  to  hire  an  outside  Lead  Appraiser  to  take  them  through  an  ARC  Class 
A,  B,  or  C  SCAMPI  to  ensure  their  preparedness  for  the  Government  SCAMPI  team. 


Table  111-17:  Example  SCAMPI-Imposed  Development  Organization  Costs 


1  Items 

|  Person-Effort  Weeks 

Calendar-Time  Weeks 

Preparation  Time 

4-6 

4-6 

Site  Readiness  Visit  Impact 

1-4 

1-2 

On-site  Visit  (3-4  projects,  approximately  3  days 
total  interview  time) 

1-2 

1-2 

POC  and  Debriefing  Time 

1 

1 

Total  Time 

7-13 

7-11 

The  site  visit  costs  are  those  associated  with  conducting  individual  interviews  (that  is,  the 
cost  of  the  interviewees’  time).  The  POC  and  debriefing  time  costs  are  those  incurred  by  the 
offeror  POC,  who  supports  the  SCAMPI  team,  coordinates  activities  with  the  company,  and 
schedules  individuals  for  interviews.  This  POC  also  prepares  individuals  before  their  inter¬ 
views  and  debriefs  the  interviewees  after  each  interview.  The  “On-site  Visit”  person-effort 
time  involves  selected  project  members  (4-5  personnel  represented  by  3^4  projects),  sup¬ 
porting  groups  (e.g.,  quality  assurance,  configuration  management),  organized  for  interviews 
in  functional  area  representative  groups  of  roughly  5  individuals  or  so,  plus  program  manager 
interviews  for  each  project.  Typically,  each  interview,  whether  individual  or  group,  will  last 
approximately  one  and  a  half  hours.  These  costs  vary  considerably  from  organization  to  or¬ 
ganization.  Of  note  are  the  hidden  costs  indirectly  of  concern  to  an  acquisition  agency.  These 
boil  down  to  disruption  of  current  activities  to  undergo  the  SCAMPI  and  could  be  a  concern, 
particularly  if  the  company  is  currently  working  on  acquisition  agency  programs  and  deliver¬ 
ables. 

Hidden  costs  include 

•  schedule  impact  on  projects  submitted  for  the  SCAMPI 
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•  project  resources  that  must  be  prepared  and  made  available  for  the  SCAMPI 

•  unplanned  schedule  accommodations  for  the  SCAMPI 

Travel  Expenses 

Costs  can  increase  if  some  contractor  staff  must  travel  to  another  site  to  accommodate  a 
SCAMPI.  Sometimes  the  projects  selected  for  the  evaluation  are  within  a  product  line  and 
division  that  may  be  at  different  locations.  While  the  SCAMPI  Method  encourages  project 
selection  within  the  same  geographical  location,  this  cannot  always  be  done  because  of  the 
development  organization’s  organizational  structure.  Development  organization  personnel 
traveling  to  accommodate  a  SCAMPI  will  not  only  be  spending  travel  funds,  their  SCAMPI- 
associated  labor  costs  will  be  greater  as  well.  Under  these  circumstances,  the  SCAMPI  team 
should  work  with  the  development  organization’s  POC  to  help  minimize  the  impact  on  those 
affected  projects. 

The  development  organization’s  preparatory  costs  are  significant:  a  period  of  four  to  six 
weeks.  The  offeror’s  operations  will  be  disrupted  by  SCAMPI  site  activities  for  approxi¬ 
mately  three  weeks  due  to  company  SCAMPI  preparation,  interview,  and  debriefing  activi¬ 
ties.  These  estimated  costs  will  change  depending  on  the  contractor  and  also  as  contractor 
familiarity  with  the  SCAMPI  process  grows  and  preparation  becomes  routine. 

Appraisal  Reuse 

The  following  paragraphs  provide  a  brief  discussion  of  efforts  being  investigated  by  the  Gov¬ 
ernment  to  reduce  the  impact  of  acquisition  agency  requirements  for  appraisals  on  new  pro¬ 
curements  and  contracts. 


Today’s  acquisition  environment  has  spawned  a  significant  interest  in  the  reuse  of  appraisal 
data.  The  concept  is  that  of  reusing  the  appraisal  data  generated  for  one  purpose  for  that  of 
another.  For  example,  if  company  X  underwent  a  SCAMPI  and  believed  that  those  results 
were  valid  for  an  upcoming  acquisition,  company  X  would  then  submit  those  results  in  lieu 
of  undergoing  another  SCAMPI.  Alternatively,  if  company  X,  as  part  of  its  overall  process 
improvement  program,  underwent  a  SCAMPI  at  its  own  expense,  it  might  then  propose  to 
use  the  results  from  that  appraisal  in  lieu  of  undergoing  an  acquisition-related  SCAMPI 
within  a  set  period  of  time.  Although  attractive  in  concept  due  to  the  potential  cost  savings 
for  both  the  Government  and  the  development  community,  details  that  would  fully  implement 
this  concept  have  not  been  fully  explored  and  approved  to  date.  The  pressure  to  meet  the  in¬ 
tent  of  Government  policy  against  the  realities  of  acquisition  schedules  and  costs  have  given 
rise  to  this  concept  and  some  acquisition  agencies  have  been  experimenting  with  various  im¬ 
plementations.  Although  the  concepts  have  been  proposed  for  a  period  of  years,  it  is  still  an 
individual  acquisition  agency’s  decision  whether  to  embrace  the  concept  and  employ  it  to  suit 
agency  needs. 
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For  source  selection,  the  following  considerations  are  recommended.  Government  program 
managers  are  encouraged  to  reuse  results  of  previous  independently-led  appraisals  that  have 
been  conducted 

•  within  two  years 

•  on  the  organizational  units  proposing  to  do  the  work 

•  using  evaluation  criteria  similar  to  CMMI  or  SW-CMM  Level  3  goals  with  one  of  the 
following  acceptable  appraisal  methods: 

-  independently  led  SCAMPI 

-  Software  Capability  Evaluation  (SCE) 

-  CMM-Based  Appraisal  for  Internal  Process  Improvement  (CBA-IPI) 

-  Software  Development  Capability  Evaluation  (SDCE) 

When  submitting  the  results  of  previous  independently  led  appraisals,  bidding  organizations 
that  have  had  appraisals  that  did  not  cover  processes  critical  to  the  project  might  be  expected 
to  have  those  processes  separately  evaluated.  Successful  offerors  might  expect  to  have  the 
Government  program  office  independently  verify  the  capabilities  of  the  organizations  prior  to 
contract  award. 

Implementing  Mechanisms  that  Support  the  Reuse  of  Appraisals 

To  reduce  costs  and  resources  associated  with  evaluating  development  capabilities,  the  DoD 
is  working  with  industry  and  Federally  Funded  Research  and  Development  Centers 
(FFRDCs)  to  provide  implementing  mechanisms  that  better  support  the  reuse  of  appraisals. 

Independently-led  appraisals:  In  responding  to  requests  for  proposals,  bidders  may  be  re¬ 
quested  or  allowed  to  submit  the  results  of  previous  appraisals  conducted  on  the  organiza¬ 
tional  unit  proposing  to  do  the  work,  as  long  as  the  organizations  have  ensured  the  “inde¬ 
pendence”  of  the  Lead  Appraiser — one  who  is  not  from  within  the  organization  being 
appraised  and  one  who  did  not  provide  the  consulting  for  guiding  the  process  improvement 
efforts. 

Government  participation  in  appraisals:  To  provide  additional  credibility  of  findings  and 
results  of  appraisals,  organizations  planning  to  compete  for  Government  contracts  may  in¬ 
volve  Government  participation  in  the  appraisal  team.  (Government  participation  might  come 
from  an  FFRDC  if  selected  by  a  Government  office  to  serve  on  behalf  of  the  Government  for 
purposes  of  the  appraisal.)  It  would  be  the  role  of  (at  least  one  of)  the  Government  partici¬ 
pants  to  sign  the  SCAMPI  Appraisal  Disclosure  Statement  (ADS),  or  similar  document,  af¬ 
firming  that  the  information  in  the  ADS  is  accurate  and  that  the  appraisal  described  in  the 
ADS  was  conducted  in  full  compliance  with  the  requirements  of  the  appraisal  method.  Any 
Government  participant  providing  this  affirmation  role  would  be  expected  to  serve  as  a  mem- 
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ber  of  the  appraisal  team  (not  an  observer),  requiring  the  participant  to  have  been  trained  in 
the  appraisal  methodology  and  model  used  for  the  appraisal. 

Registration  of  Cooperative  Government/Industry  Appraisals:  To  provide  an  additional 
mechanism  that  program  managers  might  use  in  support  of  their  source  selection  activities, 
the  DoD  is  seeking  ways  to  establish  a  process,  including  a  registration  form,  for  Cooperative 
Government/Industry  Appraisals  that  would  be  used  to  attest  that  the  appraisal  fully  complied 
with  the  specified  appraisal  method.  It  would  also  be  used  by  organizations  to  submit  the 
findings  of  previous  appraisals,  thus  reducing  requirements  for  multiple,  subsequent  apprais¬ 
als. 
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Appendix  F  Sample  Source  Selection 

Plan  and  RFP  Language 


The  following  is  source  selection  plan  language  describing  how  SCAMPI  appraisals  may  be 
used  in  source  selection.  It  is  meant  to  serve  only  as  an  example  and  so  has  been  included  “as 


is. 


99 


Conduct  Maturity  Appraisal 

Input:  Input  to  the  appraisal  will  include  methods  and  procedures  tailored  from  the  Software 
Engineering  Institute  (SEI)  Standard  CMMIsm  Appraisal  Method  for  Process  Improvement 
(SCAMPIsm)  Version  1.1.  Input  to  the  appraisal  from  the  contractor  will  include  documenta¬ 
tion  identified  in  Section  L  of  the  RFP. 


Process:  The  Government  (identify  what  unit/division)  will  conduct  a  maturity  appraisal  with 
the  overall  purpose  of  evaluating  the  offeror’s  approach  to  product  development  and  the  ma¬ 
turity  of  their  related  engineering  and  management  processes.  A  significant  portion  of  this 
evaluation  will  be  a  survey  of  contractor  processes  and  capabilities  in  accordance  with  the 
SEI  SCAMPI.  It  will  be  done  on  site  at  a  location  selected  by  the  contractor,  conducive  to 
performing  an  overall  appraisal  of  the  offeror’s  engineering  and  management  capabilities. 

Site  visits  will  occur  prior  to  the  start  of  the  proposal  evaluation.  A  SCAMPI  qualified  team 
composed  of  source  selection  evaluators  and  advisors  will  perform  the  evaluation.  The  results 
of  the  evaluation  will  be  included  as  input  to  the  Technical/Management  (pick  one  or  insert  a 
different  criterion  or  factor)  evaluation.  The  conduct  of  the  Technical/Management/etc.  Fac¬ 
tor  will  be  generally  consistent  with  other  factor  evaluations. 

Output:  The  initial  output  of  the  SCAMPI  evaluation  will  be  a  report  that  documents  the 
specific  results  of  the  appraisal.  It  will  be  provided  to  the  factor  evaluators  prior  to  comple¬ 
tion  of  their  individual  evaluations.  Evaluators  and  advisors  will  use  this  report  in  conjunc¬ 
tion  with  the  proposal  data  to  support  the  individual  strengths/weaknesses  and  risk  assess¬ 
ment  of  the  Technical/Management/etc.  Factor.  Subsequent  output  following  consensus 
evaluation  of  this  factor  will  be  the  Consensus  Evaluation  Report  and  the 
Cost/Schedule/Performance  Risk  Identification  Worksheet. 


The  following  is  source  selection  plan  language  describing  the  related  evaluation  factor,  us¬ 
ing  Management  as  an  example. 
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This  factor  includes  the  offeror’s  management  approach  to  product  development  (including 
quality  assurance,  integration,  validation,  and  verification)  and  maintenance  encompassing 
but  not  limited  to  planning,  estimating,  tracking,  and  reviewing  the  engineering  and  devel¬ 
opment  activities  requisite  to  the  program’s  success. 


The  standards  to  satisfy  this  are: 

•  The  engineering  and  development  approach  describes  the  ability  of  the  offeror  to  ade¬ 
quately  define,  plan,  track,  review,  validate  and  verily,  and  control  its  product  develop¬ 
ment  activities,  and  that  of  its  subcontractors,  requisite  to  the  program’s  success. 

•  The  product  development  resources  and  capabilities  demonstrate  the  adequacy  of  the  of¬ 
feror’s  engineering  and  management  processes  to  support  program  requirements. 

RFP  Language 

x  x  Product  Development  Capabilities 

In  support  of  the  Management  Factor  evaluation,  the  Government  intends  to  conduct  an 
evaluation  of  the  product  development  and  management  capabilities  of  the  offeror  team 
(combined  organizational  capability,  not  individual  capabilities).  The  evaluation  will  involve 
methods  and  procedures  tailored  from  the  Software  Engineering  Institute  (SEI)  Standard 
CMMIsm  Appraisal  Method  for  Process  Improvement  (SCAMPIsm)  Version  1.1  using  Capa¬ 
bility  Maturity  Model®  Integration  (CMMIsm)  for  Systems  Engineering,  Software  Engineer¬ 
ing,  and  Integrated  Product  and  Process  Development  (CMMI-SE/SW/IPPD),  Version  1.1. 
The  on-site  portion  of  this  appraisal  will  be  conducted  at  a  site  selected  by  the  offeror  and 
conducive  to  performing  an  overall  evaluation  of  the  contractor  management  and  product 
development  capabilities.  The  offeror  shall  provide  the  SCAMPI  documentation  described  in 
Attachment  A  of  this  RFP  to  support  this  evaluation.  (This  documentation  could  include  the 
Maturity  Questionnaire  and  list  of  candidate  programs  from  across  the  offeror  team,  from 
which  the  Government  will  select  three  or  more  to  evaluate.).  This  documentation  shall  be 
provided  in  the  proposal  and  will  not  be  included  in  the  page  count  limitations  for  the  pro¬ 
posal. 

After  the  SCAMPI  documentation  is  received,  the  Government  will  coordinate  a  site  visit 
with  each  offeror  to  conduct  a  SCAMPI  at  a  site  selected  by  the  offeror.  The  site  chosen 
should  be  capable  of  demonstrating  all  of  the  relevant  engineering  and  management  practices 
to  be  employed  on  this  effort.  If  the  offeror  plans  on  developing  the  product  at  more  than  one 
site,  team  members  from  the  other  development  sites  should  be  present  at  the  SCAMPI  loca¬ 
tion  to  participate  in  the  SCAMPI.  The  offeror  shall  provide,  with  the  SCAMPI-related  sub¬ 
mission  described  in  Appendix  A  [this  appendix  is  not  included  in  this  document],  a  point  of 
contact  and  phone  number  at  the  selected  site  for  the  SCAMPI  team  leader  to  coordinate  all 
SCAMPI  activities.  The  Government  will  also  communicate  details  about  the  site  visit  during 
the  coordination  process.  The  offeror  will  be  notified  of  the  projects  to  be  examined  and  any 
specific  areas  of  interest  approximately  15  working  days  prior  to  the  site  visit.  The  site  visit 
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dates  selected  by  the  Government  are  not  open  for  discussion.  The  SCAMPI  team  will  need  a 
closed  meeting  room  capable  of  accommodating  at  least  eight  people.  The  otferor  should 
have  copies  of  the  appropriate  engineering  and  management  standards,  procedures,  and/or 
operating  instructions,  and  organizational  charts  for  the  projects  being  reviewed  in  the  meet¬ 
ing  room  when  the  SCAMPI  team  arrives. 

Other  Section  L/Section  M  Example 

Section  L  -  Instructions,  Conditions,  and  Notices  to  Offerors 

General  Instructions 

a.  This  section  of  the  Instructions  for  Proposal  Preparation  (IFPP)  provides  general  guid¬ 
ance  for  preparing  proposals  as  well  as  specific  instructions  on  the  format  and  content  of 
the  proposal.  The  offeror’s  proposal  must  include  all  data  and  information  requested  by 
the  IFPP  and  must  be  submitted  in  accordance  with  (IAW)  these  instructions.  The  offer 
shall  be  compliant  with  the  requirements  as  stated  in  Statement  of  Objectives  (SOO), 
Technical  Requirements  Document  (TRD),  Contract  Data  Requirements  List  (CDRL), 
and  Model  Contract.  Non-conformance  with  the  instructions  provided  in  the  IFPP  may 
result  in  an  unfavorable  proposal  evaluation.  All  claimed  technical,  management,  per¬ 
formance  and  schedule  capabilities  to  meet  the  requirements  shall  be  realistic  and  are 
subject  to  verification  by  the  Government. 

b.  The  offeror’s  Phase  I  proposal  shall  consist  of  a  written  submittal  and  oral  presentations. 
The  offeror’s  Phase  II  proposal  shall  consist  of  all  information  provided  by  the  offeror  (as 
long  as  it  is  consistent  with  the  constraints  of  this  Request  for  Proposal  (RFP),  e.g.,  page 
limitations,  appropriate  volume)  and  is  subject  to  evaluation  as  described  in  Section  M. 

c.  The  proposal  shall  be  clear  and  concise  and  shall  include  sufficient  detail  for  effective 
evaluation  and  for  substantiating  the  validity  of  stated  claims.  The  proposal  should  not 
simply  rephrase  or  restate  the  Government’s  requirements;  rather  it  should  address  how 
the  offeror  intends  to  meet  these  requirements. 

d.  The  proposal  acceptance  period  is  specified  in  Section  A  of  the  model  contract/ 
solicitation.  The  proposal  shall  be  valid  for  a  period  of  not  less  than  270  days  from  the 
required  submission  date.  The  offeror  shall  make  a  clear  statement  in  Section  A  of  the 
proposal  documentation  volume  that  the  proposal  is  valid  until  this  date. 

e.  The  proposal  will  be  submitted  by  the  closing  date  specified  in  Section  A.  The  Govern¬ 
ment  will  conduct  an  evaluation  of  these  documents.  The  completion  and  submission  of 
these  Volumes  constitutes  the  offeror’s  acceptance  to  the  terms  and  conditions  in  this 
RFP  and  in  any  attachments  thereto.  Since  the  Government  reserves  the  right  to  award  a 
contract  without  discussions,  as  provided  by  FAR  52.215-1,  offerors  are  cautioned  to 
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consult  with  the  Contracting  Officer  before  submitting  an  offer  that  takes  exception  to 
any  term  or  condition  of  the  RfP.  Volume  due  dates  are  depicted  in  Table  III- 1 8. 

f.  The  Government  will  consider  offeror’s  exceptions  to  the  terms  and  conditions  of  this 
RFP.  Exceptions  must  be  submitted  with  Volume  IV. 

g.  The  offeror  shall  assume  that  the  Government  has  no  prior  knowledge  of  its  facilities  and 
experience,  and  will  base  its  evaluation  on  the  information  presented  in  the  offeror’s  pro¬ 
posal  and,  if  a  site  visit  is  made,  on  the  information  collected  during  the  site  visit. 

h.  Elaborate  brochures  or  documentation,  binding,  detailed  artwork,  or  other  embellish¬ 
ments  are  unnecessary  and  are  not  desired.  Similarly,  for  oral  presentations,  elaborate 
productions  are  not  desired. 

i.  IAW  FAR  Subpart  4.8  (Government  Contract  Files),  the  Government  will  retain  one  copy 
of  all  unsuccessful  proposals.  The  Government  will  destroy  the  extra  copies  of  unsuc¬ 
cessful  proposals  unless  the  offeror  specifically  requests  at  the  time  the  proposal  is  sub¬ 
mitted  that  they  be  returned. 

j.  Restrictions  on  Disclosure  and  Use  of  Proposal — A  proposal  may  include  data,  such  as 
technical  design  or  concept,  financial  data  or  management  plans,  that  the  offeror  does  not 
want  disclosed  to  the  public  for  any  purpose  or  used  by  the  Government  for  any  purpose 
other  than  evaluation  of  the  proposal  and  post-award  use.  If  the  offeror  wishes  to  restrict 
his  proposal,  it  shall  be  marked  IAW  corporate  procedures  for  marking  proprietary,  non- 
releasable  information. 

General  Information 

Point  of  Contact 

XXXXX,  the  Procuring  Contracting  Officer  (PCO),  is  the  ‘'Primary  point  of  contact,”  and 
XXXXX  is  the  “Alternate  point  of  contact”  for  this  acquisition.  Address  any  questions  or 
concerns  you  may  have  to  the  PCO.  Written  requests  for  clarification  may  be  sent  to  the  PCO 
at  the  address  located  in  Section  A,  DD  Form  1707,  Block  4,  “Issuing  Office”  of  the  model 
contract/solicitation. 

Debriefings 

All  offerors  may  request  a  debriefing  by  providing  a  written  request  to  the  PCO.  The  PCO 
will  notify  offerors  of  the  award  decision  within  3  calendar  days  after  award.  Offerors  desir¬ 
ing  a  debriefing  must  request  one  in  writing  within  3  days  after  PCO  notification.  To  the 
maximum  extent  practicable,  debriefings  will  be  conducted  within  5  days  of  receipt  of  the 
offeror’s  request.  Debriefings  will  be  conducted  IAW  AFFARS  5315.506. 
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Table  111-18:  Proposal  Organization  Phase  I  and  Volume  Due  Dates 


Vol¬ 

ume 

IFPP 

Paragraph 

Number 

Volume  Title/Section 

Hard 

Copy 

Qty* 

Soft 

Copy 

Qty* 

Date  of 
Submission 

I 

3.0, 3.1, 3.2 

Guide  to  Proposal 

Content/ Attachment  L-l 
Cross  Reference  Matrix 

3 

3 

Maximum  of  3 
pages  for 

Guide  to  Pro¬ 
posal  Content 
plus  attach¬ 
ment  L-l 

xxxxxxx 

Ila 

4.0, 4.1, 4.1.1, 
4.2.1, 4.4.1 

Mission  Capability/ 
Architecture  and  Inte¬ 
grated  Master  Plan, 
Software  Development 
Effort  Estimate 

3 

3 

Maximum  of 

30  pages 

xxxxxxx 

lib 

4.5 

Capability  Maturity 

Model  Integration 
(CMMI)  Risk  Mitigation 
Plan 

3 

3 

Maximum  of  5 
pages,  only  if 
required 

xxxxxxx 

lie 

4.6 

Mission  Capability/ 
Program  Risk 

3 

3 

Included  in  30 
pages  of  Vol¬ 
ume  II,  section 

a 

xxxxxxx 

lid 

5.0,  5.1, 5.1.1, 
5.1.2, 5.1.3,  5.2 

Mission  Capability 
(Oral) 

20 

2 

Limited  to 
charts  actually 
briefed  within 
the  three-hour 
time  limit 

Completed 
within  30 
days  of 
scheduled 
due  date  for 
proposals 

IV 

7.0,  7.1,  7.2,  7.3 
(excluding 

7. 3.4.1  and 

7. 3.4.5) 

Contract  Documentation 

3 

3 

Unlimited 
(without  pric¬ 
ing  informa¬ 
tion  until 
requested) 

All  but  the 
price  infor¬ 
mation 
XXXXXXX 

V 

8.0 

Relevant  Past 

Performance 

3 

3 

Maximum  3 
page 

introduction 

Desired  15 
days  after 
issuance  of 
RFP,  no  later 
than  30  days 
after  issuance 
of  RFP 

Volume  II  Mission  Capability  (Oral  Presentation) 

All  offerors  will  be  required  to  provide  the  Government  evaluation  team  an  oral  presentation. 
The  charts  from  this  volume  that  are  used  by  the  offeror  during  the  oral  presentation  (and  the 
videotape  of  the  oral  presentation)  will  be  used  to  evaluate  the  Mission  Capability  and  Pro¬ 
posal  Risk  factors  IAW  Section  M.  Oral  presentation  shall  address  the  following  process  ar¬ 
eas  and  the  IMP  processes  identified  in  paragraph  4.2. 
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Oral  Presentation  Requirements 

Location  and  Videotaping 

Oral  presentations  shall  be  conducted  at  the  offeror’s  facility.  Offerors  are  responsible  for 
videotaping  the  oral  presentation.  Seating  for  up  to  18  members  of  the  Government  team, 
including  evaluators  and  advisors,  will  be  required.  For  planning  purposes,  oral  presentations 
are  expected  to  be  completed  within  30  calendar  days  from  the  scheduled  due  date  for  receipt 
of  proposals.  The  Government  will  randomly  determine  the  order  in  which  the  offerors  will 
present  their  oral  presentation.  Slides  submitted  but  not  briefed  within  the  time  limit  will  not 
be  considered  for  evaluation.  The  Government  may  issue  additional,  written  questions  at  the 
conclusion  of  the  Q&A  session.  Written  responses  to  these  questions  generated  at  the  conclu¬ 
sion  of  the  orals  are  required  to  be  delivered  (in  electronic  format)  to  the  CO  at  the  address 
located  in  Section  A,  DD  Form  1707,  Block  4,  “Issuing  Office”  of  the  model  con¬ 
tract/solicitation  not  later  than  72  hours  following  completion  of  said  orals.  The  Government 
reserves  the  right  to  issue  questions  or  Evaluation  Notices  (ENs)  on  any  portion  of  the  pro¬ 
posals  (written  or  oral  presentations)  at  any  time.  The  videotape  of  the  oral  presentation  will 
receive  the  same  care  and  security  as  all  other  source  selection  material. 

Time  Limits  and  Personnel  to  Brief 

The  contractor’s  oral  presentation  shall  not  exceed  a  total  of  three  hours.  A  one  and  a  half 
hour  lunch  break  will  be  scheduled  after  three  hours  of  oral  presentation.  The  Government 
team  plans  to  caucus  on  the  morning  session  during  the  lunch  break  to  review  and  consoli¬ 
date  questions,  so  offerors  are  requested  to  arrange  for  a  working  lunch  and  a  private  meeting 
room.  One  fifteen-minute  break  will  be  scheduled  during  the  three  hours  of  oral  presentation. 
Breaks  and  lunch  will  not  count  against  the  time  limit.  The  Government  may  ask  minor  clari¬ 
fication  questions  during  the  three-hour  oral  presentation,  however,  if  the  offeror  determines 
that  the  answer  will  require  a  lengthy  response,  they  can  defer  their  answer  to  the  Q&A  ses¬ 
sion  at  the  end  of  the  three-hour  presentation.  A  two-hour  question  and  answer  (Q&A)  ses¬ 
sion  will  follow  the  caucus,  and  will  also  be  videotaped.  The  Q&A  session  is  not  planned  to 
exceed  two  hours.  The  Government  will  evaluate  ONLY  information  recorded  on  the  video¬ 
tape  and  corresponding  slide  hardcopies. 


a.  The  five-hour  time  limit  will  commence  at  the  Government’s  direction.  The  filming  of 
the  oral  presentation  will  be  terminated  at  the  completion  of  the  five-hour  time  limit,  or 
the  conclusion  of  the  offeror’s  presentation,  whichever  occurs  first.  Questions  and  an¬ 
swers  will  be  videotaped  and  will  be  evaluated. 

b.  The  offeror’s  Program  Manager  and/or  other  key  personnel  shall  give  the  oral  presenta¬ 
tion.  Each  presenter  shall  give  a  one-slide  presentation  of  their  personal  qualifica¬ 
tions/responsibilities  as  they  relate  to  the  XXXX  effort.  An  individual  authorized  to  obli¬ 
gate  the  company  contractually  shall  also  be  present.  Additional  personnel  may  also  be 
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present  at  the  oral  presentation  to  address  specific  questions  as  deemed  necessary  by  their 
Program  Manager. 

Format  and  Copies 

a.  Oral  Presentations  shall  be  in  briefing  format  and  will  be  provided  at  the  start  of  the  oral 
presentation.  Offerors  shall  not  submit  annotated  briefing  charts.  Facing  page  text  shall 
contain  the  RFP  Matrix  Section  L  and  M  evaluation  criteria  that  applies  to  each  chart. 

The  offeror  shall  begin  by  introducing  the  oral  presentation  team  by  name,  position  held, 
role  each  person  will  have  after  contract  award,  and  company  affiliation.  The  offeror 
shall  use  the  oral  presentation  to  explain  its  understanding  and  approach  to  enable  com¬ 
plete  evaluation  of  the  offeror’s  capability  to  provide  the  products  and  services  as  re¬ 
quired  by  the  Request  for  Proposal  (RFP).  The  offeror  shall  demonstrate  its  plans  to  meet 
the  stated  requirements  or  goals  and  show  that  it  possesses  the  necessary  understanding 
and  expertise  to  successfully  accomplish  the  proposed  work.  The  offeror  shall  identify 
any  additional  types  of  information  it  believes  are  needed  in  the  performance  of  the  con¬ 
tract.  NO  PRICE  information  shall  be  included  in  the  oral  presentation  or  briefing  charts. 


b.  Prior  to  the  Government  team’s  departure,  the  offeror  shall  provide  three  (3)  copies  of  an 
unedited  VHS  video  recording  of  their  oral  presentation  as  presented  to  the  Government 
and  of  the  question  and  answer  (Q&A)  period.  One  copy  will  serve  as  the  master  record 
of  the  oral  presentation.  The  2nd  and  3rd  copies  will  be  used  for  viewing  by  the  evaluation 
team.  The  Government  reserves  the  right  to  duplicate  the  videotapes,  if  required. 

Oral  Presentation  Content 

Using  the  IMP  as  a  roadmap,  the  offeror  shall  provide  a  summary  of  their  architecture  ap¬ 
proach  and  details  of  their  integrated  processes.  The  offeror  should  include  those  risk  areas 
that  they  plan  to  work  during  the  contract  period  of  performance.  Order  of  the  presentation  is 
left  to  the  discretion  of  the  offeror.  The  Government  plans  to  issue  Evaluation  Notices,  as  ap¬ 
plicable,  prior  to  the  oral  presentation.  Offerors  shall  reflect  their  responses  to  the  Evaluation 
Notices  in  the  content  of  their  oral  presentation.  Charts  presented  during  the  oral  presenta¬ 
tions  that  address  Evaluation  Notices  shall  be  so  noted  in  the  title  of  the  respective  chart. 

The  offeror  shall  provide  an  overview  of  the  architectural  methodology  and  identify  the 
planned  set  of  architecture  views.  The  offeror  shall  describe  the  processes  to  identify,  define, 
analyze,  verify,  document,  and  test  all  functional  and  derived  requirements.  Describe  the 
process  to  ensure  that  user,  developer,  and  maintainer  needs  are  satisfied  in  the  system  re¬ 
quirements.  Explain  how  new  requirements  are  assessed  for  their  impact  on  cost,  schedule, 
and  performance.  Describe  the  software  development  effort  estimation  process,  including 
linkage  to  systems  engineering  and  change  processes,  standard  methodologies  and  models, 
and  how  the  initial  software  effort  estimates  will  be  updated  throughout  the  system  life  cycle. 
Describe  the  configuration  management  processes,  with  reference  to  both  internally  and  ex¬ 
ternally  generated  changes  including  the  process  to  provide  feedback  and  recommended 
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changes  to  the  tailored  Database  Design  Description.  Describe  the  process  for  managing 
changes  to  the  organization’s  system  and  software  development  activities.  Describe  the  fac¬ 
tors  considered  in  determining  how  the  overall  syste  development  groups  and  subgroups 
are  organized.  Describe  how  internal  communications  will  be  managed  to  include  the  conflict 
resolution  process  among  internal  development  groups.  Describe  how  you  intend  to  manage 
the  systems  engineering  processes  and  the  critical  dependencies  between  development  activi¬ 
ties.  Explain  how  you  plan  to  provide  the  Government  with  insight  into  development  and 
maintenance  efforts.  Describe  the  process  for  building  reliability  and  maintainability  into  the 
system  to  minimize  total  cost  of  ownership.  Explain  how  you  will  provide  user  documenta¬ 
tion  to  include,  but  not  limited  to,  systems  operations  and  maintenance.  Describe  the  process 
for  selecting,  integrating,  upgrading,  and  managing  ND1  products  in  the  system  architecture 
throughout  the  system  life  cycle.  Describe  the  process  for  managing  the  impacts  of  obsoles¬ 
cence  of  NDI  products  on  the  system  architecture.  Explain  the  process  for  identifying  and 
managing  software  defects  throughout  the  XXXX  program  system  life  cycle.  Describe  the 
approach  to  value-added  engineering  to  include  identifying,  evaluating,  selecting,  and  inte¬ 
grating  new  technologies  into  the  XXX  system  architecture.  Describe  the  process  for  system 
integration  and  test.  Describe  the  process  to  integrate  the  test  activities  with  a  Combined  Test 
Force  (CTF).  Explain  how  test  plans,  test  procedures,  and  test  cases  are  developed,  docu¬ 
mented,  reviewed,  and  controlled.  Explain  which  metrics  you  propose  to  collect  and  how  the 
metrics  will  be  computed,  analyzed,  and  reported.  Describe  the  approach  to  providing  system 
training  to  users,  operators,  and  maintainers.  Describe  the  approach  to  managing  Contractor 
Logistics  Support  (CLS)  for  the  XXXX  program  after  IOC. 

Section  M  -  Evaluation  Factors  for  Award 

Evaluation  Factors  and  Subfactors  and  their  Relative  Order  of 

importance 

Award  will  be  made  to  the  offeror  proposing  a  program  most  advantageous  to  the  Govern¬ 
ment  based  on  an  integrated  assessment  of  the  evaluation  factors  described  below.  The  first 
three  evaluation  factors  (Mission  Capability,  Past  Performance,  and  Proposal  Risk)  are  equal 
in  importance,  and  each  is  more  important  than  the  Cost/Price  Factor.  Within  the  Mission  Ca¬ 
pability  and  Proposal  Risk  factors,  the  Subfactors  are  in  descending  order  of  importance.  In 
accordance  with  FAR  15.403(e),  the  evaluation  factors  other  than  cost,  when  combined,  are 
significantly  more  important  than  cost;  however,  cost  will  contribute  substantially  to  the  se¬ 
lection  decision. 

Factor  1:  Mission  Capability 
Subfactor  1:  Architecture 
Subfactor  2:  Integrated  Processes 

Factor  2:  Past  Performance 

Factor  3:  Proposal  Risk 
Subfactor  1:  Architecture 
Subfactor  2:  Integrated  Processes 
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Factor  4:  Cost/Price 


Mission  Capability  Factor 

The  offeror’s  written  proposal,  oral  presentation,  and  the  Subcontracting  Plan  will  be  used  to 
evaluate  the  Mission  Capability  Factor.  In  general,  the  evaluation  will  assess  the  offeror’s 
understanding  of  requirements  and  whether  the  proposed  approach  is  sound,  within  budget 
constraints  in  Section  L,  and  consistent  with  their  proposed  schedule. 

Positive  consideration  may  be  given  to  credible  plans  for  achieving  the  following  areas 
within  budget  constraints  listed  in  descending  order  of  importance: 

1 .  early  implementation  of  IOC 

2.  early  implementation  of  Data  Warehouse  (DW) 

3 .  early  implementation  of  FOC 

Subfactor  1,  Architecture 

The  Government  will  evaluate  the  offeror’s  proposed  system  architecture  and  technical  ap¬ 
proach  to  determine  compliance  and  consistency  with  users’  requirements  as  defined  in  the 
Technical  Requirements  Document  (TRD)  and  Statement  of  Objectives  (SOO).  The  subfactor 
will  assess  whether  the  offeror’s  approach  is  achievable  within  budget  constraints  set  forth  in 
Section  L  and  is  at  a  minimum  a  flexible,  effective,  evolvable  approach  that 

•  implements  the  architecture  approach  and  technical  solutions  within  the  offeror’s  pro¬ 
posed  Plan  bounded  by  the  budget  constraints  in  Section  L 

•  implements  the  application  of  data  independence  techniques  that  separate  logic  from  the 
data 

•  provides  for  quick  and  efficient  adding  or  changing  of  source  system  interfaces 

•  provides  a  flexible  interface  that  allows  users  and  customer  systems  to  control  queries 
and  display  formats 

•  provides  metadata  and  business  rules  to  the  user  for  data  identification,  storage,  and 
query 

•  implements  an  effective  and  flexible  data  architecture  and  describes  how  the  design  ac¬ 
commodates  changes  efficiently 

•  efficiently  integrates  proposed  tools 

•  evolves  with  technical  change  and  functional  requirements  over  time 

•  demonstrates  knowledge  of  the  Defense  Transportation  System  required  to  effectively 
develop  XXXX 

•  provides  systems  security 
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Subfactor  2,  Integrated  Processes 

The  Government  will  evaluate  the  offeror’s  integrated  processes  and,  if  required,  the  CMMI 
Risk  Mitigation  plan,  to  ensure  XXXXX  program  activities  and  products  (e.g.  hardware, 
software,  and  logistics  elements)  provide  an  overarching,  executable,  and  integrated  solution 
set  consistent  with  their  proposed  architecture  and  design  that  satisfies  both  the  SOO  objec¬ 
tives  and  TRD  requirements  throughout  the  system  life  cycle.  The  Government  will  evaluate 
the  integrated  processes  to  include  system  engineering,  program  management  including 
Earned  Value  Management  (EVM),  configuration  management,  integration,  test,  installation 
and  logistics  support  (including  training).  In  addition,  the  Government  will  evaluate  the  of¬ 
feror’s  Subcontracting  Plan  to  determine  that  the  plan,  at  a  minimum,  specifies  subcontract¬ 
ing  goals  for  small  disadvantaged  business  in  accordance  with  FAR  52.219-9,  DFARs 
252.219-7003  and  DFARs  252.219-7004;  and  addresses  how  those  goals  will  be  met  and 
sustained.  The  Government  has  set  the  Small  Disadvantaged  goal  at  5%.  The  subfactor  will 
assess  whether  the  offeror’s  approach  provides,  at  a  minimum 

•  a  System  Engineering  process  that  provides  the  Government  insight  into  the  design  and 
test  as  it  progresses  to  IOC  and  demonstrates  effective  team  mg  with  the  Government  to 
ensure  functional  requirements  are  satisfactorily  met  in  the  most  efficient  manner 

•  an  evolutionary  acquisition  approach  using  spiral  development 

•  an  EVM  process  and  tools  that  ensure  current,  accurate,  readily  available  information 

•  an  integrated  risk  management  process  to  identify,  communicate  to  the  Government, 
analyze,  and  mitigate  program  risks 

•  a  partnership  approach  in  which  the  contractor  and  Government  work  together,  making 
daily  program  decisions  to  effectively  manage  the  program 

•  a  process  to  identify,  evaluate,  select,  integrate,  and  maintain  COTS  products,  including 
the  criteria  that  each  product  must  meet  before  it  is  considered  and  is  consistent  with  the 
proposed  architecture 

•  a  process  to  identify,  evaluate,  select,  and  integrate  new  technologies  and  assess  their 
impact  on  functional  requirements,  system  architecture  and  Life-Cycle-Cost 

•  an  effective  subcontractor  management  process 

•  a  software  development  process  that  derives  and  allocates  requirements  to  the  design  and 
ensures  that  the  requirements  are  verifiable;  manages  the  initial  baseline  and  require¬ 
ments  changes,  including  traceability  from  the  requirements  to  the  design  and  from  the 
design  to  the  requirements;  manages  multiple,  interdependent  software  configuration 
baselines;  ensures  that  documentation  is  current,  accurate,  complete,  and  readily  avail¬ 
able,  including  but  not  limited  to  architecture,  design  implementation,  metadata,  and 
business  rules 

Past  Performance  Factor 

Under  the  Past  Performance  factor,  the  Performance  Confidence  Assessment  represents  the 
evaluation  of  an  offeror’s  and  associated  key  or  major  subcontractors’,  teaming  partners’,  and 
joint  venture  partners’  present  and  past  work  records  to  assess  the  Government’s  confidence 
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the  offeror  will  successfully  perform  as  proposed.  The  Government  will  evaluate  the  offeror’s 
and  all  key  or  major  subcontractors’,  teaming  partners’,  and  joint  venture  partners’  demon¬ 
strated  record  of  contract  compliance  in  supplying  products  and  services  that  meet  user’s 
needs,  including  cost  and  schedule.  The  Past  Performance  Evaluation  is  accomplished  by  re¬ 
viewing  the  aspects  of  an  offeror’s  and  all  key  or  major  subcontractors’,  teaming  partners’, 
and  joint  venture  partners’  present  and  recent  past  performance,  focusing  on  and  targeting 
performance  that  is  relevant  to  the  Mission  Capability  subfactors  and  the  requirements  of  the 
solicitation.  The  Government  may  consider  as  relevant  efforts  performed  for  agencies  of  fed¬ 
eral,  state,  or  local  Governments  and  commercial  customers.  As  a  result  of  an  analysis  of  this 
past/current  work  history,  each  offeror  will  receive  a  Performance  Confidence  Assessment, 
which  is  the  rating  for  the  Past  Performance  factor.  Although  the  past  performance  evaluation 
focuses  on  performance  that  is  relevant  to  the  Mission  Capability  subfactors,  the  resulting 
Performance  Confidence  Assessment  is  made  at  the  factor  level  and  represents  an  overall 
evaluation  of  the  likelihood  of  successful  contractor  performance.  The  Government  will 
evaluate  current  and  past  (within  the  last  three  years)  performance  to  determine  the  Govern¬ 
ment’s  confidence  in  each  offeror’s  ability  to  successfully  perform  the  XXXX  effort.  The 
Government  will  assess  the  performance  and  relevancy  of  each  offeror’s  work  on  current  and 
past  contracts,  both  Government  and  Commercial.  This  information  may  include  data  on  ef¬ 
forts  performed  by  other  divisions,  critical  subcontractors,  or  teaming  contractors,  if  such 
resources  will  be  brought  to  bear  or  significantly  influence  the  performance  of  the  proposed 
effort.  Past  performance  for  other  divisions,  critical  subcontractors,  or  teaming  contractors 
must  be  for  the  same  type  of  effort  as  is  proposed  for  XXXX.  The  PRAG  will  then  go  on  to 
determine  relevancy  in  accordance  with  the  following  criteria: 

To  be  considered  at  least  somewhat  relevant,  prime’s  past  performance  efforts  must  have 
been  performed  by  the  same  division  and  location  within  the  past  three  (3)  years.  Subcon¬ 
tractors  will  be  assessed  as  either  relevant  or  not  relevant.  In  order  for  a  subcontractor’s  effort 
to  be  considered  Relevant,  the  contract  must  have  been  performed  at  the  same  divi¬ 
sion/location,  within  the  past  three  years,  and  must  have  been  the  same  type  effort  proposed 
for  the  XXXX  program.  The  Government  will  then  assess  only  contract  efforts  considered  to 
be  somewhat  relevant,  relevant,  or  very  relevant  for  the  prime’s  and  relevant  for  subcontrac¬ 
tors  in  the  determination  of  the  Confidence  rating.  Relevancy  for  prime  contractor’s  efforts 
will  be  based  on  the  following  criteria: 

1 .  development  and  implementation  of  a  large  (at  least  25  GB,  50  entities,  and  250  attrib¬ 
utes  for  Very  Relevant  and  Relevant  and  at  least  20  GB,  50  entities,  and  250  attributes 
for  Somewhat  Relevant)  database  integration  effort  using  data  modeling  techniques  and 
state-of-the-art  database  tools  to  design  and  maintain  the  database 

2.  development  of  any  of  the  DOD  systems  that  are  part  of  the  XXXX  as  listed  in  Attach¬ 
ment  L-7 

3 .  software-intensive  system  development  contract  for  $  1 00M  or  greater  total  contract 
value  within  the  past  5  years,  (3  consecutive  years  of  performance) 
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4.  similar  cost  type  contract  (this  does  not  include  time  and  material,  or  labor  hour  con¬ 
tracts) 

5.  selection  and  integration  of  COTS  with  developed  code 

6.  software  development/integration  effort  valued  at  $5M  per  year  or  greater 

7.  past  work  involving  Contractor  Logistics  Support 

8.  developing  web-based,  end-user  applications 

9.  past  work  involved  subcontractor  interaction,  or  work  with  team  partners  or  joint  ven¬ 
ture  partners 

In  order  to  be  considered  Very  Relevant,  must  meet  the  applicable  portion  of  number  1  plus 
2,  3,  4,  and  at  least  3  of  the  remaining  5  criteria  listed  above.  To  be  considered  Relevant,  must 
meet  applicable  portion  of  number  1,  either  2  or  3,  and  at  least  4  of  the  6  remaining  criteria 
listed  above.  To  be  considered  Somewhat  Relevant,  must  meet  the  applicable  portion  of 
number  1,  plus  at  least  4  of  the  remaining  8  criteria  listed  above. 

When  an  offeror’s  relevant  performance  record  indicates  performance  problems,  the  Gov¬ 
ernment  will  consider  the  number  and  severity  of  the  problems  and  the  appropriateness  and 
effectiveness  of  any  corrective  actions  taken  (not  just  planned  or  promised).  The  Government 
may  review  more  recent  contracts  or  performance  evaluations  to  ensure  corrective  actions 
have  been  implemented  and  to  evaluate  their  effectiveness. 

Each  offeror  will  receive  one  of  the  ratings  described  in  AFFARS  5315.305  for  the  Past  Per¬ 
formance  factor.  Offerors  without  a  record  of  relevant  past  performance  or  for  whom  infor¬ 
mation  on  past  performance  is  not  available  will  not  be  evaluated  favorably  or  unfavorably 
on  past  performance  and,  as  a  result,  will  receive  a  “Neutral/Unknown  Confidence”  rating  for 
the  Past  Performance  factor. 

More  recent  and  relevant  performance  may  have  a  greater  impact  on  the  Performance  Confi¬ 
dence  Assessment  than  less  recent  or  less  relevant  effort.  A  strong  record  of  relevant  past  per¬ 
formance  may  be  considered  more  advantageous  to  the  Government  than  a  “Neu¬ 
tral/Unknown  Confidence”  rating.  Likewise,  a  recent  relevant  record  of  favorable 
performance  may  receive  a  higher  confidence  rating  and  be  considered  more  favorably  than  a 
less  recent  relevant  record  of  favorable  performance. 

Past  performance  information  may  be  obtained  through  the  Contractor  Performance  Assess¬ 
ment  Reporting  Systems  (CPARS),  similar  systems  of  other  Government  departments  and 
agencies,  questionnaires  tailored  to  the  circumstances  of  this  acquisition,  Defense  Contract 
Management  Agency  (DCMA)  channels,  interviews  with  program  managers  and  contracting 
officers  and  other  sources  known  to  the  Government,  including  commercial  sources.  Offerors 
are  to  note  that,  in  conducting  this  assessment,  the  Government  reserves  the  right  to  use  both 
data  provided  by  each  offeror  and  data  obtained  from  other  sources. 
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In  addition  to  the  material  evaluated  for  Phase  I,  the  Phase  II  evaluations  will  assess  any  up¬ 
dates  to  those  efforts. 

Proposal  Risk  Factor 

a.  The  content  under  the  Mission  Capability  subfactors  in  paragraph  3.0  will  apply  to  the 
assessment  of  the  Proposal  Risk  subfactors  and  will  consider  the  information  contained 
in  the  technical  proposal,  cost  proposal,  and  the  results  of  the  Executive  In-Plant  Review, 
if  conducted.  The  Proposal  Risk  assessment  focuses  on  the  risks  and  weaknesses  associ¬ 
ated  with  an  offeror’s  proposed  approach  and  includes  an  assessment  of  the  potential  for 
disruption  of  schedule,  degradation  of  performance,  and  the  need  for  increased  Govern¬ 
ment  oversight,  as  well  as  the  likelihood  of  unsuccessful  contract  performance.  For  each 
identified  risk,  the  assessment  also  addresses  the  offeror’s  proposal  for  mitigating  the  risk 
and  why  that  approach  is  or  is  not  manageable. 

b.  A  Performance  Confidence  Assessment  will  be  assigned  to  the  Past  Performance  factor. 
Performance  confidence  represents  the  Government’s  confidence  in  an  offeror’s  ability  to 
successfully  perform  as  proposed  based  on  an  assessment  of  the  offeror’s  present  and 
past  work  record.  Each  offeror  will  receive  one  of  the  ratings  described  in  AFFARS 
5315.305(a)(2)(ii)  for  the  Past  Performance  factor.  As  a  result  of  an  integrated  analysis  of 
those  risks  and  strengths  identified,  each  offeror  will  receive  a  single  integrated  Perform¬ 
ance  Confidence  Assessment.  The  Performance  Confidence  Assessment  will  be  the  sole 
rating  for  the  Past  Performance  factor. 

c.  A  proposal  risk  rating  will  be  assigned  to  each  Mission  Capability  subfactor  under  the 
proposal  risk  factor.  Proposal  risk  represents  the  risks  identified  with  an  offeror’s  pro¬ 
posed  approach  as  it  relates  to  the  evaluation  criteria  and  solicitation  requirements.  Each 
subfactor  under  the  Proposal  Risk  factor  will  receive  one  of  the  Proposal  Risk  ratings  de¬ 
scribed  in  AFFARS  5315.305(a)(3)(ii). 

d.  Cost/Price  will  be  evaluated  as  described  in  AFFARS  5315.305(a)(l)(i). 

e.  After  the  SSET  has  completed  their  evaluation  of  the  written  and  oral  proposals,  includ¬ 
ing  past  performance  information,  the  results  of  each  offeror’s  interim  evaluation  will  be 
provided.  By  1500  hours  ET  of  the  day  following  receipt  of  interim  ratings  an  offeror 
may  request  an  opportunity  to  conduct  a  one-hour  rebuttal  to  the  SSAC  and  the  SSA  col¬ 
lectively.  The  offeror’s  presentation  will  be  limited  to  the  information  provided  to  the 
SSET  in  determining  the  interim  ratings  provided.  Location  and  time  for  these  rebuttal 
activities  is  TBD.  For  Phase  I  this  rebuttal  activity  would  take  place  before  the  request  for 
Complete/Cost  Proposals  are  issued.  For  Phase  II  these  rebuttals  would  take  place  after 
receipt  of  the  initial  Phase  II  cost  proposal  and  technical  deliverables  but  before  request 
(if  required)  for  Final  Proposal  Revisions. 
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f.  When  the  integrated  assessment  of  all  aspects  of  the  evaluation  is  accomplished,  includ¬ 
ing  changes,  if  any,  resulting  from  rebuttal  activity  set  forth  in  paragraph  e  above,  the 
color  ratings,  performance  confidence  assessment,  proposal  risk  ratings  and  price  evalua¬ 
tion  will  be  considered  in  the  order  of  priority  stated  in  paragraph  2.0.  Any  one  of  these 
considerations  can  influence  the  SSA’s  decision. 


To  be  determined:  Section  M  Standards  for  Evaluation. 
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Appendix  G  OSD  Software  Evaluation 

Policy  Memorandum 


ACQUISITION  AND 
TECHNOLOGY 


THE  UNDER  SECRETARY  OF  DEFENSE 

3010  DEFENSE  PENTAGON 
WASHINGTON,  DC  20301-3010 


2  6  OCT  1999 


MEMORANDUM  FOR  COMPONENT  ACQUISITION  EXECUTIVES 

DIRECTOR  OF  BALLISTIC  MISSILE  DEFENSE  ORGAN  (ZATION 

SUBJECT:  Software  Evaluations  for  AC  AT  I  Programs 


It  is  DoD  policy  that  software  systems  be  designed  and  developed  based  upon  software 
engineering  principles.  This  includes  the  selection  of  contractors  with  the  domain  experience  in 
developing  comparable  software  systems,  a  successful  past  performance  record,  and  a 
demonstrable  mature  software  development  capability  and  process.  It  also  requires  a  software 
measurement  process  to  plan  and  track  the  software  program,  and  to  assess  and  improve  the 
development  process  and  associated  software  product. 

Software  development  and  performance  is  an  integral  component  of  advanced  defense 
systems.  Accordingly,  it  will  be  a  technical  requirement  for  contract  that  each  coutiacioi 
performing  software  development  or  upgrade(s)  for  use  in  an  ACAT I  program  will  undergo  an 
evaluation,  using  either  the  tools  developed  by  the  Software  Engineering  Institute  (SEI),  or  those 
approved  by  the  DoD  Components  and  the  DUSD(S&T). 

At  a  minimum,  full  compliance  with  SEI  Capability  Maturity  Model  Level  3,  oj  its 
equivalent  level  in  an  approved  evaluation  tool,  is  the  Department's  goal.  However,  if  (he 
prospective  contractor  does  not  meet  full  compliance,  a  risk  mitigation  plan  and  schedule  must 
be  prepared  that  will  describe,  in  detail,  actions  that  will  be  taken  to  remove  deficiencies 
uncovered  in  the  evaluation  process  and  must  be  provided  to  the  Program  Manager  for  approval. 
The  Deputy  Under  Secretary  of  Defense  (Science  &  Technology)  will  define  Level  3 
equivalence  for  approved  evaluation  tools.  The  evaluation  will  he  performed  on  the  business  unit 
proposed  to  perform  the  work.  The  reuse  of  existing  evaluation  results  performed  within  a  two- 
year  period  prior  to  the  date  of  the  government  solicitation  is  encouraged. 

This  policy  is  effective  immediately  and  will  be  incorporated  into  the  current  DoD  5000 
series  rewrite. 


(jb.  S.  Gansler 
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Glossary 


The  MDD  glossary  defines  many,  but  not  all,  terms  used  in  this  document.  The  following 
additional  sources  for  terms  and  definitions  should  be  considered  supplementary  to  the  MDD 
glossary: 

•  CMMI  model  glossary  and  terminology 

•  ARC  glossary 

Terms  that  are  particularly  significant  to  this  document  are  duplicated  from  the  model  docu¬ 
ment  or  ARC  for  convenience. 

An  observation  extracted  from  data  collected  during  an  appraisal 
that  has  been  determined  by  the  appraisal  team  to  be  (a)  worded 
appropriately,  (b)  based  on  information  seen  or  heard,  (c)  relevant 
to  the  appraisal  reference  model  being  used,  (d)  significant  such 
that  it  can  be  classified  as  a  strength,  weakness,  or  alternative 
practice,  and  (e)  not  redundant  with  other  observations,  [arc  vi.i] 

An  oral  or  written  statement  confirming  or  supporting  implemen¬ 
tation  of  a  CMMI  model  specific  practice  or  generic  practice.  Af¬ 
firmations  are  usually  provided  by  the  implementers  of  the  practice 
and/or  internal  or  external  customers,  but  may  also  include  other 
stakeholders  (e.g.,  managers,  suppliers),  [derived  from  MDD  method  over¬ 
view]  Interview  responses  are  examples  of  face-to-face  affirmations. 
Alternative  forms  of  affirmations  could  include  presentations  or 
demonstrations  of  a  tool  or  mechanism  as  it  relates  to  implementa¬ 
tion  of  a  CMMI  model  practice,  [derived  from  MDD  PII  appendix  B] 

alternative  practice  A  practice  that  is  a  substitute  for  one  or  more  generic  or  specific 

practices  contained  in  the  CMMI  model  that  achieves  an  equiva¬ 
lent  effect  toward  satisfying  the  goal  associated  with  the  practices. 
Alternative  practices  are  not  necessarily  one-for-one  replacements 
for  the  generic  or  specific  practices,  [arc  vl.l  and  cmmi  model  glossary] 

appraisal  An  examination  of  one  or  more  processes  by  a  trained  team  of  pro¬ 

fessionals  using  an  appraisal  reference  model  as  the  basis  for  de- 


accurate 

observation 


affirmation 
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termining,  as  a  minimum,  strengths  and  weaknesses,  [arc  vi.i] 


Appraisal 
Disclosure 
Statement  (ADS) 


appraisal  findings 


appraisal  input 


appraisal  method 
class 


appraisal  modes  of 
usage 


appraisal 

objectives 

appraisal  output 


appraisal 

participants 

appraisal 

rating 


A  summary  statement  describing  the  ratings  generated  as  outputs 
of  the  appraisal,  and  the  conditions  and  constraints  under  which 
the  appraisal  was  performed.  The  ADS  should  be  used  for  public 
disclosures  of  maturity  level  or  capability  level  ratings  so  they  can 
be  interpreted  accurately,  [local] 

The  results  of  an  appraisal  that  identify  the  most  important  issues, 
problems,  or  opportunities  for  process  improvement  within  the 
appraisal  scope.  Appraisal  findings  are  inferences  drawn  from 
valid  observations.  [CMMI  model  glossary  and  ARC  vl.l] 

The  collection  of  appraisal  information  required  before  data  col¬ 
lection  can  commence,  [iso  98C  and  arc  vi.i] 


A  family  of  appraisal  methods  that  satisfy  a  defined  subset  of  re¬ 
quirements  in  the  Appraisal  Requirements  for  CMMI  (ARC).  These 
classes  are  defined  so  as  to  align  with  typical  usage  modes  of  ap¬ 
praisal  methods,  [derived  from  ARC  vl.O,  CMMI  model  glossary  and  ARC  vl.l] 


The  contexts  in  which  an  appraisal  method  might  be  utilized.  Ap¬ 
praisal  modes  of  usage  identified  for  the  SCAMPI  method  include 
internal  process  improvement,  supplier  selection,  and  process 
monitoring. 


The  desired  outcome(s)  of  an  appraisal  process.  [ARCvi.i] 


All  of  the  tangible  results  from  an  appraisal  (see  “appraisal  re¬ 
cord”).  [ISO  98C  and  ARC  vl.l] 

Members  of  the  organizational  unit  who  participate  in  providing 
information  during  the  appraisal.  [CMMI  model  glossary  and  arc  vl.l] 


The  value  assigned  by  an  appraisal  team  to  either  (a)  a  CMMI  goal 
or  process  area,  (b)  the  capability  level  of  a  process  area,  or  (c)  the 
maturity  level  of  an  organizational  unit.  The  rating  is  determined 
by  enacting  the  defined  rating  process  for  the  appraisal  method 
being  employed.  [CMMI  model  glossary  and  ARC  vl.l] 
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appraisal  record 


appraisal 
reference  model 

appraisal  scope 


appraisal  sponsor 


appraisal  tailoring 


appraisal 
team  leader 


artifact 


assessment 


capability 

evaluation 


An  orderly,  documented  collection  of  information  that  is  pertinent 
to  the  appraisal  and  adds  to  the  understanding  and  verification  of 
the  appraisal  findings  and  ratings  generated,  [derived  from  iso  98C  and 
ARCvl.l] 

The  CMMI  model  to  which  an  appraisal  team  correlates  imple¬ 
mented  process  activities.  [CMMI  model  glossary  and  ARC  vl.l] 

The  definition  of  the  boundaries  of  the  appraisal  encompassing  the 
organizational  limits  and  the  CMMI  model  limits  within  which  the 
processes  to  be  investigated  operate,  [derived  from  cmmi  model  glossary, 
ISO  98C  and  ARC  vl.l] 

The  individual,  internal  or  external  to  the  organization  being  appraised, 
who  requires  the  appraisal  to  be  performed,  and  provides  financial  or 
Other  resources  to  cany  it  out.  [derived  fiom  ISO  98C  and  ARC  vl.l] 

Selection  of  options  within  the  appraisal  method  for  use  in  a  spe¬ 
cific  instance.  The  intent  of  tailoring  is  to  assist  an  organization  in 
aligning  application  of  the  method  with  its  business  needs  and  ob¬ 
jectives.  [CMMI  model  glossary  and  ARC  vl.l] 


The  person  who  leads  the  activities  of  an  appraisal  and  has  satis¬ 
fied  the  qualification  criteria  for  experience,  knowledge,  and  skills 
defined  by  the  appraisal  method.  [ARC  vl.l] 

A  tangible  form  of  objective  evidence  indicative  of  work  being  per¬ 
formed  that  is  a  direct  or  indirect  result  of  implementing  a  CMMI 
model  practice.  (See  “direct  artifact”  and  “indirect  artifact.”) 


An  appraisal  that  an  organization  does  to  and  for  itself  for  the  pur¬ 
poses  of  process  improvement.  [ARC  vl.l] 

An  appraisal  by  a  trained  team  of  professionals  used  as  a  discrimi¬ 
nator  to  select  suppliers,  for  contract  monitoring,  or  for  incentives. 
Evaluations  are  used  to  gain  insight  into  the  process  capability  of  a 
supplier  organization  and  are  intended  to  help  decision  makers 
make  better  acquisition  decisions,  improve  subcontractor  perform¬ 
ance,  and  provide  insight  to  a  purchasing  organization,  [arc  vl.l] 
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consensus 


consolidation 


corroboration 


coverage 


coverage  criteria 


data  collection 
session 


direct  artifact 


discovery-based 

appraisal 


A  method  of  decision  making  that  allows  team  members  to  de¬ 
velop  a  common  basis  of  understanding  and  develop  general 
agreement  concerning  a  decision  that  all  team  members  are  willing 
to  support.  [ARC  v  1.1] 

The  activity  of  collecting  and  summarizing  the  information  pro¬ 
vided  into  a  manageable  set  of  data  to  (a)  determine  the  extent  to 
which  the  data  are  corroborated  and  cover  the  areas  being  investi¬ 
gated,  (b)  determine  the  data’s  sufficiency  for  making  judgments, 
and  (c)  revise  the  data-gathering  plan  as  necessary  to  achieve  this 
sufficiency.  [ARCvi.i] 

The  extent  to  which  enough  data  has  been  gathered  to  confirm  that 
an  observation  is  acceptable  for  use  by  an  appraisal  team.  [ARCvi.i] 
In  SCAMPI,  corroboration  is  obtained  through  method  require¬ 
ments  for  the  collection  of  practice  implementation  indicators  of 
multiple  types  (see  “practice  implementation  indicator”). 

The  extent  to  which  objective  evidence  gathered  addresses  a  model 
component  within  the  scope  of  an  appraisal.  [ARCvi.i] 

The  specific  criterion  that  must  be  satisfied  in  order  for  coverage 
to  be  claimed,  [arc  vi.i] 

An  activity  during  which  information  that  will  later  be  used  as  the 
basis  for  observation  formulation  or  corroboration  is  gathered. 

Data  collection  sessions  (or  activities)  include  the  administration 
and/or  analysis  of  instruments,  document  review,  interviews,  and 
presentations.  [ARCvi.i] 

The  tangible  outputs  resulting  directly  from  implementation  of  a 
specific  or  generic  practice.  An  integral  part  of  verifying  practice 
implementation.  May  be  explicitly  stated  or  implied  by  the  practice 
statement  or  associated  informative  material,  [mdd  method  overview] 

An  appraisal  in  which  limited  objective  evidence  is  provided  by 
the  appraised  organization  prior  to  the  appraisal,  and  the  appraisal 
team  must  probe  and  uncover  a  majority  of  the  objective  evidence 
necessary  to  obtain  sufficient  coverage  of  CMMI  model  practices. 
Discovery-based  appraisals  typically  involve  substantially  greater 
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appraisal  team  effort  than  verification-based  appraisals,  in  which 
much  of  the  objective  evidence  is  provided  by  the  appraised  orga¬ 
nization.  (See  verification-based  appraisal  for  contrast.) 

document  A  collection  of  data,  regardless  of  the  medium  on  which  it  is  re¬ 

corded,  that  generally  has  permanence  and  can  be  read  by  humans 
or  machines.  [ARC  vi.i]  In  SCAMPI,  documents  are  work  products 
reflecting  the  implementation  of  one  or  more  model  practices.  This 
typically  includes  work  products  such  as  organizational  policies, 
procedures,  and  implementation-level  work  products.  Documents 
may  be  available  in  hardcopy,  softcopy,  or  accessible  via 
hyperlinks  in  a  web-based  environment,  [derived  from  mdd  method  over- 

view] 

Evaluation  Notices  Exchanges  with  offerors  for  the  purpose  of  clarification,  communica¬ 
tion,  or  discussion.  Evaluation  Notices  that  result  from  deficiencies  in 
the  offeror’s  proposal  are  clearly  identified  to  the  offeror  as  deficien- 

I  2  II  ‘  Ullillli  cies.  I  ||;|| '  |  IP  18gp[  lit  7  ;!;  /  '  ff  I 

findings  The  conclusions  of  an  assessment,  evaluation,  audit,  or  review  that 

identify  the  most  important  issues,  problems,  or  opportunities 
within  the  appraisal  scope.  Findings  include,  at  a  minimum, 
strengths  and  weaknesses  based  on  valid  observations,  [arc  vi.i] 

A  technique  to  prioritize  appraisal  team  effort  based  on  the  con¬ 
tinuous  collection  and  consolidation  of  appraisal  data,  and  moni¬ 
toring  of  progress  toward  achieving  sufficient  coverage  of  CMMI 
model  practices.  Appraisal  resources  are  targeted  toward  those  ar¬ 
eas  for  which  further  investigation  is  needed  to  collect  additional 
data  or  verify  the  collected  set  of  objective  evidence,  [derived  from 
MDD  method  overview] 

An  interview  technique  in  which  individuals  and/or  groups  of  in¬ 
dividuals  are  asked  to  participate  in  validation  of  preliminity  otPi|; 
sesrvatiqps  prepred  i>y  a;SCA||tPI  team  bausiedpn  da%  obtained  iit  7 
previous  interview  sessions  and  documentation  reviews. 

fully  implemented  A  practice  characterization  value  assigned  to  a  process  instantia- 

(Fl)  tion  when  (1)  direct  artifacts  are  present  and  judged  to  be  appro¬ 

priate,  (2)  at  least  one  indirect  artifact  and/or  affirmation  exists  to 
confirm  the  implementation,  and  (3)  no  substantial  weaknesses  are 


focus  groups 


focused 

investigation 
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noted.  [MDD  3.7.2] 


indirect  artifact 


instantiation 


instruments 


internal  process 
improvement  (IPI) 


interviews 


largely  implemented 
(LI) 


An  artifact  that  is  a  consequence  of  performing  a  specific  or  ge¬ 
neric  practice  or  that  substantiate  its  implementation,  but  which  is 
not  the  purpose  for  which  the  practice  is  performed.  This  indicator 
type  is  especially  useful  when  there  may  be  doubts  about  whether 
the  intent  of  the  practice  has  been  met  (e.g.,  a  work  product  exists 
but  there  is  no  indication  of  where  it  came  from,  who  worked  to 
develop  it,  Or  how  it  is  USed).  [MDD  method  overview] 

For  practices  implemented  by  projects,  each  project;  for  practices 
implemented  organization-wide,  the  instance. 

Artifacts  used  in  an  appraisal  for  the  collection  and  presentation  of 
data  (e.g.,  questionnaires,  organizational  unit  information  packets). 
[arc  vi.i]  In  SCAMPI,  instruments  are  used  to  collect  written  in¬ 
formation  relative  to  the  organizational  unit’s  implementation  of 
CMMI  model  practices.  This  can  include  assets  such  as  question¬ 
naires,  surveys,  or  an  organizational  mapping  of  CMMI  model 
practices  to  its  corresponding  processes. 

An  appraisal  mode  of  usage  in  which  organizations  appraise  inter¬ 
nal  processes,  generally  to  either  baseline  their  process  capability, 
to  establish  or  update  a  process  improvement  program,  or  to  meas¬ 
ure  progress  in  implementing  such  a  program,  [derived  from  mdd 

method  overview] 


A  meeting  of  appraisal  team  members  with  appraisal  participants 
for  the  purpose  of  gathering  information  relative  to  work  processes 
in  place,  [arc  vi.i]  In  SCAMPI,  this  includes  face-to-face  interac¬ 
tion  with  those  implementing  or  using  the  processes  within  the 
organizational  unit.  Interviews  are  typically  held  with  various 
groups  or  individuals,  such  as  project  leaders,  managers,  and  prac¬ 
titioners.  A  combination  of  formal  and  informal  interviews  may  be 
held  and  interview  scripts  or  exploratory  questions  developed  to 
elicit  the  information  needed. 

A  practice  characterization  value  assigned  to  a  process  instantia¬ 
tion  when  (1)  direct  artifacts  are  present  and  judged  to  be  appro¬ 
priate,  (2)  at  least  one  indirect  artifact  and/or  affirmation  exists  to 
confirm  the  implementation,  and  (3)  one  or  more  weaknesses  are 
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noted.  [MDD  3.7.2] 


lead  appraiser 

mini-team 

not  implemented 
(Nl) 

objective  evidence 

observation 

organizational  unit 


partially 

implemented  (PI) 


A  person  who  has  achieved  recognition  from  an  authorizing  body 
to  perform  as  an  appraisal  team  leader  for  a  particular  appraisal 
method.  [ARCvi.i] 

See  “process  area  mini-team.” 

A  practice  characterization  value  assigned  when  the  appraisal  team 
determines  insufficient  objective  evidence  exists  to  state  that  the 
practice  is  implemented.  That  is,  the  criteria  for  assigning  a  value 
of  Fully  Implemented  (FI),  Largely  Implemented  (LI),  or  Partially 
Implemented  (PI)  are  not  satisfied,  [local] 


Qualitative  or  quantitative  information,  records,  or  statements  of 
fact  pertaining  to  the  characteristics  of  an  item  or  service  or  to  the 
existence  and  implementation  of  a  process  element,  which  is  based 
on  observation,  measurement,  or  test  and  which  can  be  verified. 
[CMMI  model  glossary,  ISO  98C  and  ARC  vl.l]  In  SCAMPI,  sources  of  ob¬ 
jective  evidence  include  instruments,  presentations,  documents, 
and  interviews. 


A  written  record  that  represents  the  appraisal  team  members’  un¬ 
derstanding  of  information  either  seen  or  heard  during  the  ap¬ 
praisal  data  collection  activities.  The  written  record  may  take  the 
form  of  a  statement  or  may  take  alternative  forms  as  long  as  the 
information  content  is  preserved.  [CMMi  model  glossary ,  arc  vi .  i] 


That  part  of  an  organization  that  is  the  subject  of  an  appraisal  (also 
known  as  the  organizational  scope  of  the  appraisal).  An  organiza¬ 
tional  unit  deploys  one  or  more  processes  that  have  a  coherent 
process  context  and  operates  within  a  coherent  set  of  business  ob¬ 
jectives.  An  organizational  unit  is  typically  part  of  a  larger  organi¬ 
zation,  although  in  a  small  organization,  the  organizational  unit 
may  be  the  whole  organization.  [Derived  from  CMMI  model  glossary,  ISO  98C 
and  ARC  vl.l] 

A  practice  characterization  value  assigned  to  a  process  instantia¬ 
tion  when  (1)  direct  artifacts  are  absent  or  judged  to  be  inadequate, 
(2)  artifacts  or  affirmations  suggest  that  some  aspects  of  the  prac¬ 
tice  are  implemented,  and  (3)  weaknesses  have  been  documented. 
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[MDD  3.7.2] 


practice 

characterization 


practice 
implementation 
indicator  (Pll) 


preliminary  findings 


presentations 


process  area  mini¬ 
team 


process  context 


The  assignment  of  a  value  describing  the  extent  to  which  a  CMMI 
model  practice  is  implemented,  used  as  a  mechanism  to  reach  ap¬ 
praisal  team  consensus.  The  range  of  values  for  practice  charac¬ 
terization  values  include  Fully  Implemented  (FI),  Largely  Imple¬ 
mented  (LI),  Partially  Implemented  (PI),  and  Not  Implemented 
(NI).  Practice  characterization  values  are  assigned  to  each  CMMI 
model  practice  for  each  process  instantiation  within  the  appraisal 
scope,  and  aggregated  to  the  organizational  unit  level,  [local] 

An  objective  attribute  or  characteristic  used  as  a  “footprint”  to 
verify  the  conduct  of  an  activity  or  implementation  of  a  CMMI 
model  specific  or  generic  practice.  Types  of  practice  implementa¬ 
tion  indicators  include  direct  artifacts,  indirect  artifacts,  and  af¬ 
firmations.  [derived  from  15504-9  and  MDD  method  overview] 


Initial  findings  created  by  an  appraisal  team  after  consolidating 
and  synthesizing  valid  observations  to  provide  the  findings  to  ap¬ 
praisal  participants  for  validation  of  accuracy,  [derived  from  arc  vi.i] 


In  SCAMPI,  a  source  of  objective  evidence  that  includes  informa¬ 
tion  prepared  by  the  organization  and  delivered  visually  or  ver¬ 
bally  to  the  appraisal  team  to  aid  in  understanding  the  organiza¬ 
tional  processes  and  implementation  of  CMMI  model  practices. 
This  typically  includes  such  mechanisms  as  orientation  or  over¬ 
view  briefings,  and  demonstrations  of  tools  or  capabilities,  [derived 

from  MDD  method  overview] 


A  subset  of  the  appraisal  team  members,  typically  two  or  three', 
assigned  primary  responsibility  for  collection  of  sufficient  ap¬ 
praisal  data  to  ensure  coverage  of  their  assigned  reference  model 
process  areas,  [local] 


The  set  of  factors  documented  in  the  appraisal  input  that  influences 
the  judgment  and  comparability  of  appraisal  ratings.  These  include, 
but  are  not  limited  to,  (a)  the  size  of  the  organizational  unit  to  be 
appraised,  (b)  the  demographics  of  the  organizational  unit,  (c)  the 
application  domain  of  the  products  or  services,  (d)  the  size,  critical¬ 
ity,  and  complexity  of  the  products  or  services,  and  (e)  the  quality 
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characteristics  of  the  products  or  services.  [CMMi  model  glossary] 


process  monitoring 


process  profile 

Quick  Lodk 


rating 

satisfied 

Source  Selection 
Authority  (SSA) 


An  appraisal  mode  of  usage  in  which  appraisals  are  used  to  moni¬ 
tor  process  implementation  (for  example,  after  contract  award  by 
serving  as  an  input  for  an  incentive/award  fee  decision  or  a  risk 
management  plan).  The  appraisal  results  are  used  to  help  the  spon¬ 
soring  organization  tailor  its  contract  or  process  monitoring  efforts 
by  allowing  it  to  prioritize  efforts  based  on  the  observed  strengths 
and  weaknesses  of  the  organization’s  processes.  This  usage  mode 
focuses  on  a  long-term  teaming  relationship  between  the  sponsor¬ 
ing  organization  and  the  development  organization  (buyer  and 

supplier),  [derived  from  MDD  method  overview] 


The  set  of  goal  ratings  assigned  to  the  process  areas  in  the  scope  of 
the  appraisal.  In  CMMI,  also  known  as  the  process  area  profile. 

[derived  from  IS098c  and  ARC  vl.l] 

An  optional  process  used  to  provide  a  quick  macroscopic  review 
of  proposal  responsiveness  to  the  Request  For  Proposal  require¬ 
ments.  Results  of  the  Quick  Look  can  be  used  to  determine  if  suf¬ 
ficient  proposal  content  is  available  to  complete  the  initial  evalua¬ 
tion.  The  Qpick  Look  results  can  also  support  t§|  determination  of 
competitive  range  or  proposal  responsiveness  if  that  determination 
is  performed  before  the  start  of  the  initial  evaluation.  In  most  cases 
the  determination  of  competitive  range  or  proposal  responsiveness 
is  performed  after  completion  of  the  initial  evaluation. 

(See  “appraisal  rating.”)  [CMMI  model  glossary  and  ARC  vl.l] 


Rating  given  to  a  goal  when  the  aggregate  of  valid  observations 
and  associated  findings  does  not  negatively  impact  achievement  of 
the  goal.  Rating  given  to  a  process  area  when  all  of  its  goals  are 
rated  “satisfied.”  [ARCvi.i] 


The  individual  responsible  for  the  conduct  of  the  government 
source  selection  (acquisition)  process.  In  an  acquisition  application 
of  the  SCAMPI  Method,  the  SSA  is  the  sponsor.  The  SSA  is  the 
final  arbiter  on  the  use  of  SCAMPI,  approves  how  the  SCAMPI 
appraisal  results  will  influence  the  award  decision,  and  makes  the 
award  decision.  (See  Source  Selection  Advisoiy  Council  and 
Source  Selection  Advisory  Board.)  [derived  from  sce  V3.o  implementation 
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Source  Selection 
Advisory  Council 
(SSAC) 


Source  Selection 
Evaluation  Board 
(SSEB) 


strength 


sufficient  data 
coverage 

supplier  selection 


tailoring 

valid  observation 


verification-based 

appraisal 


Guide  for  Supplier  Selection] 

The  SSAC  is  chartered  by  the  sponsoring  organization  (acquisition 
agency)  with  collecting  and  analyzing  the  evaluations  of  each  of¬ 
feror.  This  group  performs  risk  assessment  activities.  This  is  the 
only  group  permitted  to  compare  the  SCAMPI  appraisal  results 
(strengths  and  weaknesses)  of  the  offerors  against  one  another.  The 
SSAC  may  recommend  to  the  sponsor  how  the  SCAMPI  findings 
will  be  incorporated  into  the  award  decision  at  the  pre-RFP  release 

briefing,  [derived  from  SCE  V3.0  Implementation  Guide  for  Supplier  Selection] 

This  is  the  government  group  that  evaluates  the  offerors’  proposals 
against  defined  evaluation  standards  in  an  acquisition  application 
of  SCAMPI.  It  develops  the  evaluation  standards  and  receives  ap¬ 
proval  to  use  them  from  the  sponsor  before  the  issuance  of  the 
RFP.  It  also  performs  risk  identification  tasks.  The  SSEB  is  usu¬ 
ally  organized  into  technical  and  cost  teams  important  to  the  award 

decision,  [derived  from  SGE  V3.0  Implementation  Guide  for  Supplier  Selection] 

Exemplary  or  noteworthy  implementation  of  a  CMMI  model  prac- 
tice.  [CMMI  model  glossary  and  ARC  vl.l] 


A  determination  that  the  coverage  requirements  have  been  met. 
See  “coverage”  and  “coverage  criteria.”  [ARCvi.i] 

An  appraisal  mode  of  usage  in  which  appraisal  results  are  used  as 
a  high  value  discriminator  to  select  suppliers.  The  results  are  used 
in  characterizing  the  process-related  risk  of  awarding  a  contract  to 
a  supplier,  [derived  from  MDD  method  overview] 


See  “appraisal  tailoring.”  [ARCvi.i] 

An  observation  that  the  appraisal  team  members  agree  is  (a)  accu¬ 
rate,  (b)  corroborated,  and  (c)  consistent  with  other  valid  observa¬ 
tions.  [ARCvi.i] 

An  appraisal  in  which  the  focus  of  the  appraisal  team  is  on  verify¬ 
ing  the  set  of  objective  evidence  provided  by  the  appraised  organi¬ 
zation  in  advance  of  the  appraisal,  in  order  to  reduce  the  amount  of 
probing  and  discovery  of  objective  evidence  during  the  appraisal 
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on-site  period.  (See  discovery-based  appraisal  for  contrast.) 


weakness 


The  ineffective,  or  lack  of,  implementation  of  one  or  more  CMMI 
model  practices.  [CMMI  model  glossary  and  ARC  vl.l] 
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